I have a set of JSON files stored in a folder; the folder structure is as follows:
Main_folder
----F1
-----I1
------X.json
-----I2
-------X.json
.
.
.
.
.
.
In all, the main folder contains 36 subfolders which in turn contain 10 subfolders each having a JSON file.
The JSON file is a dictionary of a lists X={'a':[contains 73 items each of which is a dictionary that represents a 2x3 matrix],'b':[contains 73 items each of which is a dictionary]}
I want to read these JSON files, get the 2x3 matrices and store them in .npy
files.
My current code does that but it ends up saving the same npy file multiple times in the folders.
For example, X_0.npy
should be stored once in Main_folder/F1/I1
, but that same file is being saved in other folders also.
import os, json,codecs
import numpy as np
from pathlib import Path
def read_pose_from_json(json_file):
#read json file and obtain 73x6 pose array
with open(json_file) as f1:
json1_str=f1.read()
json1_data=json.loads(json1_str)
data=json1_data.copy()
l1=data['world_coordinate']
pose_list=[]
for i in range(len(l1)):
X=list(l1[i].values())[0]
X=np.array(X).ravel()
pose_list.append(X)
return np.array(pose_list)
path_to_json='blender_files/'
paths=[]
paths1=[]
#For all json files ,read them obtain pose array(73x6) and store it in a npy file so that it can
#later be accesed
for (dirpath,dirnames,filenames) in os.walk(path_to_json):
paths+=[os.path.join(dirpath,file) for file in filenames if file.startswith('bones_') and file.endswith('.json')]
for i in range(len(paths)):
pose_array=read_pose_from_json(paths[i])
new_pose_file=Path(paths[i]).stem+".npy"
new_pose_file=os.path.join(dirpath,new_pose_file)
with open(new_pose_file,'wb') as f:
np.save(f,pose_array)
What steps can I take to optimize this code, and ensure that redundant copies of the same file are not created?
I have enclosed a screenshot which shows what the code is currently doing:
In this image inside Model23/image7
there are multiple npy files being saved.
I was able to resolve the issue,I moved the loop
for i in range(len(pathfiles)
outside the loop that recursively looks in all folders and sub-folders. so for now the correct code is
use os.walk() to list all json files and store them in list path
then iterate through paths list and obtain matrices and save them
If anyone else can suggest a better solution,that would be appreciated.
npz
and not multiple. \$\endgroup\$pathlib
:files = Path('blender_files').glob('*/*/bones_*.json')
\$\endgroup\$bones_20
example has 10 items? \$\endgroup\$