Questions tagged [multiprocessing]
Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system.
90
questions
2
votes
0
answers
21
views
Brand generator script using multiprocessing
I have two indices in Elasticsearch which are brands, and categories, and I have a very ...
1
vote
0
answers
16
views
TCP socket management using multiprocessing
I wrote a Python code for a device that generates an hotspot and communicates with an app using a TCP socket. The communication is managed by a process that opens a socket and uses two threads to send ...
0
votes
1
answer
32
views
Optimise pandas code using def(x) and multiprocessing
I am trying to cleanup a dataframe (and the code) in Pandas.
I used to use the modin library before but I have since switched from Win 10 to Win 11 and none of <...
0
votes
0
answers
53
views
Efficient way to build random undirected graphs without self loops given total number of graph nodes and associated node degree
I wrote an algorithm to build 1000 different random graphs given number of nodes and node degree. Node degree is maintained between the random graphs. I am building these random graphs to input to a ...
2
votes
1
answer
97
views
My code creates a soundboard that can assign a sound to a key on a keyboard while listening to keyboard strokes in parallel
My code works but I am looking for an efficient way to implement the idea with out having to repeat the same code block inside the function Recin() (record Input function is stated in the code) in ...
0
votes
1
answer
56
views
Showcase difference multiprocessing and threading in Python
This week I dived into the issue of speeding up my program by either threading or multiprocessing. I did not understand well the difference between the two, but that became quite clear when I got the ...
0
votes
0
answers
57
views
Running an Excel Macro from a Child Process Using Multiprocessing Cannot Access Excel Object in Main Process
First, some background on what I am actually trying to do here: I am trying to open an Excel Workbook from Python, then run certain macros within that workbook, but with a timeout feature, so that, if ...
2
votes
1
answer
71
views
Multithread or multiprocess [closed]
This is my code:
...
1
vote
0
answers
37
views
Thread-safe running mean and variance
In one of my personal python libraries I have a custom class used for computing the running average and variance of a stream of numbers:
...
3
votes
0
answers
42
views
Nearest neighbor search algorithm of quantization
I'm doing some research on Nearest Neighbor Search of quantization, where running speed is very important. As I've just started using Python, I wonder if the following code for multi-processes can be ...
4
votes
0
answers
161
views
Subclass of Python's multiprocessing.Pool which allows progress reporting
For context, the whole of the project code can be found here. This question was created specifically for the progress.py file.
The goal behind it is to allow ...
2
votes
1
answer
97
views
Multi-core OpenCV denoising
I have ~40,000 JPEG images from the Kaggle melanoma classification competition. I created the following functions to denoise the images:
...
4
votes
0
answers
319
views
PyQt load images in background process
I am building an application that needs to load 100-200 images (really only limited by performance) and display them to the user in a sort of gallery. A good analogy would simply be Google Images. In ...
3
votes
1
answer
75
views
benchmarking requests to localhost
I am writing a benchmarking tool from scratch in Python.
However I can't get the performance of other benchmarking tools like wrk or ...
6
votes
1
answer
2k
views
Compute the outputs of a generator in parallel
I am processing an unknown "length" of generator object.
I have to keep things "lazy" because of memory management.
The processing is compute heavy, so writing it multiproc style ...
1
vote
0
answers
38
views
Python/Mongo - Generating dummy data
I wrote below piece of code generating dummy data loaded to MongoDB.
2 issues :
1.customer class has subscriber class so it nested with one level customer--> subsciber.
I think i'm not doing right:
...
3
votes
0
answers
382
views
Assigning parallel workers to predict from keras model using concurrent.futures
I am working on a Reinforcement learning project, where I have to gather a lot of data using a TensorFlow model. During the data gathering, the weights of the model do not change. So, I am using ...
2
votes
1
answer
46
views
Hardware driven data processing and ploting in need of better control flow
This is code for a measurement setup that receives a steady stream of UDP data, finds the trigger in one channel and operates on the data in the other channel to enhance the signal and remove noise, ...
2
votes
0
answers
19
views
Implementation of Policy Gradient Reward Design paper
I've implemented the first experiment from the Reward Design via Online Gradient Ascent paper. I don't have any specific concerns, but it's my first time using multiprocessing or doing reinforcement ...
-4
votes
1
answer
163
views
How can I reduce the TIme Complexity. of the python program
I have 7000 CSV files that I want to do operation on. but the problem is that it takes to much time like it takes 13 seconds to process these files.
What I need to know is which operation take more ...
3
votes
1
answer
3k
views
python flask spawn jobs with multiprocessing
I have a python flask app that waits for requests from user app and than spawns a process with job based on the request it receives.
It keeps the status and queue of the jobs in memory.
The requests ...
5
votes
3
answers
121
views
Analysing a Huge Codebase with Python
I've written a Python program to analyse a huge C++ code base in excess of millions of lines of code. The job of the program is simply to search for all C and C++ style comments and extract all the ...
3
votes
1
answer
84
views
Divide and Conquer Password Bruteforcer
My program brute-forces a password. The password is a string composed of a key and a four digit numeric code. The key is known so we are basically brute-forcing between 0000 through to 9999
An example ...
4
votes
2
answers
212
views
Class with multiple workers
I am building a tool that interacts with a batched stream of incoming data. This data needs to be processed and the result returned. To split up the work I have created a class that has inbound (...
1
vote
1
answer
94
views
Parallel image segmentation [closed]
Follow up question Image processing using Python OOP library
I want to use multiprocessing to analyze several images in parallel:
...
-2
votes
1
answer
48
views
Python code refactor to use multiprocessing [closed]
I have written some Python code that opens files one by one, does some work, and writes data onto the filesystem. I figured, I want to make use of my cores.I've refactored the program into this.
<...
2
votes
1
answer
26
views
Join ArcGIS tables in Python of plant greenness in response to prior month's rainfall
I am using Python 3.8 with ArcGIS Pro 2.5. This code creates a joined table where I match values for plant greenness in the current month with rainfall X number of months into the past. Each loop ...
4
votes
1
answer
80
views
Data segmentation and optimization for Multi-threaded/Multi-core use (In Python)
I have written a very basic algorithm, that counts the amount of times sub-string appears in a string.
The algorithm is given:
seq - a string sequence
k - length of a sub-string
L - a window of a ...
4
votes
0
answers
66
views
Monitored parallel mapping across a Pandas dataframe
This code maps a function over a dataset in parallel in a monitored fashion, and returns the result in the variable out.
...
5
votes
1
answer
90
views
Create 1% Sample Using Multiprocessing in Python
I'm trying to process a large dataset (300GB, myfile.txt in the script) line by line using multiprocessing. I want to define a 1% random sample based one variable (contained in unique_ids_final.txt). ...
3
votes
1
answer
85
views
Plot timings for a range of inputs MkII
To celebrate questions getting more reputation points per upvote, I revisited my previous question, Plot timings for a range of inputs.
Since the time of that original question (more than two years!),...
3
votes
2
answers
1k
views
hash image url and if valid store image url
I am trying to get all image urls which are valid and unique.To make sure i wont end up using the same image again i calculate hash of the image and store it in a text file and read the text file ...
5
votes
1
answer
145
views
Gamblers ruin, parallel compute trials
Context
The following script is a quick implementation of the gambler's ruin problem. For each given upper bound on the number of rounds in a game max_iter, ...
3
votes
1
answer
50
views
Monitor a web page for when a desired price is met
I have a code that keeps checking a webpage with a price on it and when the price meets a number I set it to, the thing is purchased. I am looking for ways to increase the speed of it.
I have added ...
3
votes
1
answer
252
views
Simple sharding of delimited files to more sophisticated
A while back I came across the issue of having a large delimited file where I wanted to simply parallelize my python code across each line of the file. However, doing so all at once either took up too ...
11
votes
2
answers
781
views
Parallelized for loop in Bash
I am using a Bash script to execute a Python script multiple times. In order to speed up the execution, I would like to execute these (independent) processes in parallel. The code below does so:
<...
1
vote
0
answers
612
views
First readers-writers problem using a single condition
For the first readers-writers problem, in which readers can access a given resource simultaneously, is the following Python3 solution correct?
Problem parameters:
One set of data is shared among a ...
2
votes
0
answers
1k
views
Image processing using multiprocessing in Python
I have an image stack. I am trying to use multiprocessing to do some process on each image in stack to get new image, and then bitwise_or with the old image.
Since ...
5
votes
0
answers
142
views
A Parallel Processing Template for Divide & Conquer Problems
I’ve written a program for solving a problem using standard single-threaded code. However, it looks like it could be recast as a multi-threaded problem using divide & conquer. Since this is a ...
10
votes
1
answer
1k
views
Python 3.7 UltimateBruteforcer
So I made a bruteforce program for sha-X hashes and MD5 using wordlists in Python.
I know Python is probably one of the worst languages for this, but it's only for learning purposes.
I think this was ...
1
vote
0
answers
49
views
Python process-subprocess communication
I have two processes. First (main) one is "dealing" with user stuff. Second one is doing some internal stuff and should update its internal state according to user actions. So I've created two pipes. ...
14
votes
2
answers
721
views
Translate nucleic acid sequence into its corresponding amino acid sequence
Goal of the program
The goal of the program is to translate a nucleic acid sequence into its corresponding amino acid sequence. The nucleic sequences
have to be formatted in a specific format called <...
5
votes
2
answers
300
views
Implementing lock within a Python script
I've written a script in Python using multiprocessing to handle multiple process at the same time and make the scraping process faster. I've used locking within it to prevent two processes from ...
2
votes
2
answers
1k
views
Extract name, address and phone number from some web pages using multiprocessing
I've written a script in Python using the multiprocessing module to scrape values from web pages (one page per subprocess). As I'm very new to ...
3
votes
0
answers
432
views
Using Perl's fork() to parallelize processing
I use bash scripts to process last names through a Perl program, and I parallelize the execution like this, by name initials:
...
3
votes
1
answer
66
views
GET multiple data frames using external module's function that uses requests
Purpose
The point of this mini project is to quickly gather data from a website's API and combine the collected DataFrames into a "master" DataFrame with all of the stocks I am interested in looking ...
9
votes
2
answers
4k
views
Python worker pattern for multithreading
I'm building a script where I pass in a text file of hostnames, and a text file of commands and each command will then be run against each host.
To prevent this getting out of hand I'm also passing ...
2
votes
1
answer
66
views
Count features in a file, using parallel code
When processing large files (say for parsing or performing computations based on the contents), we want to be able to use multiple processes to perform the job faster.
In my case I wanted to count ...
2
votes
1
answer
4k
views
Passing shared memory array to worker pool in Python
I'm working with big 2d numpy arrays, and a function that I need to iterate over each row of these arrays.
To speed things up, I've implemented parallel processing ...
3
votes
0
answers
32
views
Architecture for a parsing tool
PostgreSQL, Python 3.5, SQLAlchemy
I have a main database which has a journals table containing 30k+ journals. For each journal I have a RSS feed URL, using which I can parse and get new articles. ...