Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [multiprocessing]

Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system.

Filter by
Sorted by
Tagged with
3 votes
1 answer
84 views

Designing Multiprocess signaling and communication

Originally I meant to discuss my entire project on stepper motors for Raspberry Pi. But I see that's out of scope. If you'd like to provide high level feedback, welcome to do so in github. The lack of ...
juanmf's user avatar
  • 151
1 vote
2 answers
64 views

bucketsort with multiprocessing

I want to compare the procedure time for each number of process. I made 5000000 numbers in input.txt in random order. I read it and do bucket sorting with ...
seokkyu's user avatar
  • 11
2 votes
0 answers
165 views

Point addition on secp256k1. Exploring other alternatives for practical purposes

...
Knm's user avatar
  • 21
0 votes
0 answers
89 views

Multiple classes getting the output of a parallel process performing heavy computations

I have a python script with different classes performing various tasks. There is one class that performs heavy computations that are needed by other classes. I am using multiprocessing in order to do ...
firion's user avatar
  • 111
4 votes
2 answers
2k views

Multi-threading Class in C++

I wrote a C++ class which can take a vector of functions and their respective arguments and execute them in parallel and return a vector of results from its respective functions. ...
Darth-CodeX's user avatar
2 votes
2 answers
522 views

Code for asynchronous data stream processing with multistage pipelines

I am trying to write some code for processing streams of asynchronous data from multiple sources (Producer Nodes), process them ...
Haaput's user avatar
  • 23
0 votes
1 answer
140 views

Training a Reinforcement Learning algorithm

Problem: I'm trying to reduce the computation time and optimize specific functions of a Reinforcement Learning algorithm in the training phase and observed that one block of code takes too much ...
Student's user avatar
4 votes
1 answer
283 views

Polyominoes generator

This is my first Python program. It is not very user friendly, but it works. I used some code snippets and tricks I found in various places. And generous people at Stack Overflow helped me with ...
Dmitry Arch's user avatar
4 votes
1 answer
546 views

Prime factorization algorithm by using the multiprocessing module of Python

I may introduce a Python code of prime factorization which is from my personal project that I'm working on. ...
MYUN's user avatar
  • 59
1 vote
0 answers
53 views

Process to continuously delete resources

I have created a function that creates a multiprocessing.Process to continuously delete "resources" (instances from a certain class) from my application. When I kill one of these resources, ...
AAB07's user avatar
  • 11
1 vote
1 answer
138 views

Timeit multithreading/multiprocessing C++

Given a Timeit method which runs for n times the provided function, which approach between multithreading and multiprocessing should be better to speed up the execution of all iterations and avoid the ...
BloomShell's user avatar
0 votes
1 answer
59 views

streaming / low memory groupby in julia

I am working on a groupby-aggregation function that will work without RAM overflow issues. Essentially, I want it to run as fast as possible, while not necessarily loading the entire data structure ...
chase's user avatar
  • 123
0 votes
1 answer
52 views

Treat a list by generating a dataframe and sending the data to function via multiprocessing

To collect the list with the data from an API, I need to do these steps: ...
Digital Farmer's user avatar
2 votes
0 answers
29 views

Brand generator script using multiprocessing

I have two indices in Elasticsearch which are brands, and categories, and I have a very ...
Xus's user avatar
  • 183
1 vote
0 answers
655 views

TCP socket management using multiprocessing

I wrote a Python code for a device that generates an hotspot and communicates with an app using a TCP socket. The communication is managed by a process that opens a socket and uses two threads to send ...
firion's user avatar
  • 111
1 vote
1 answer
123 views

Efficient way to build random undirected graphs without self loops given total number of graph nodes and associated node degree

I wrote an algorithm to build 1000 different random graphs given number of nodes and node degree. Node degree is maintained between the random graphs. I am building these random graphs to input to a ...
skynaive's user avatar
  • 172
2 votes
1 answer
510 views

My code creates a soundboard that can assign a sound to a key on a keyboard while listening to keyboard strokes in parallel

My code works but I am looking for an efficient way to implement the idea with out having to repeat the same code block inside the function Recin() (record Input function is stated in the code) in ...
Learner's user avatar
  • 23
0 votes
1 answer
68 views

Showcase difference multiprocessing and threading in Python

This week I dived into the issue of speeding up my program by either threading or multiprocessing. I did not understand well the difference between the two, but that became quite clear when I got the ...
Bruno Vermeulen's user avatar
2 votes
1 answer
123 views

Multithread or multiprocess [closed]

This is my code: ...
teneji's user avatar
  • 21
1 vote
0 answers
48 views

Thread-safe running mean and variance

In one of my personal python libraries I have a custom class used for computing the running average and variance of a stream of numbers: ...
Mate de Vita's user avatar
3 votes
0 answers
49 views

Nearest neighbor search algorithm of quantization

I'm doing some research on Nearest Neighbor Search of quantization, where running speed is very important. As I've just started using Python, I wonder if the following code for multi-processes can be ...
klion's user avatar
  • 31
4 votes
0 answers
457 views

Subclass of Python's multiprocessing.Pool which allows progress reporting

For context, the whole of the project code can be found here. This question was created specifically for the progress.py file. The goal behind it is to allow ...
gabe appleton's user avatar
2 votes
1 answer
309 views

Multi-core OpenCV denoising

I have ~40,000 JPEG images from the Kaggle melanoma classification competition. I created the following functions to denoise the images: ...
Yehuda's user avatar
  • 121
4 votes
0 answers
1k views

PyQt load images in background process

I am building an application that needs to load 100-200 images (really only limited by performance) and display them to the user in a sort of gallery. A good analogy would simply be Google Images. In ...
Spencer's user avatar
  • 261
3 votes
1 answer
259 views

benchmarking requests to localhost

I am writing a benchmarking tool from scratch in Python. However I can't get the performance of other benchmarking tools like wrk or ...
Masudul Hasan Masud's user avatar
6 votes
1 answer
6k views

Compute the outputs of a generator in parallel

I am processing an unknown "length" of generator object. I have to keep things "lazy" because of memory management. The processing is compute heavy, so writing it multiproc style ...
Potter A's user avatar
1 vote
0 answers
140 views

Python/Mongo - Generating dummy data

I wrote below piece of code generating dummy data loaded to MongoDB. 2 issues : 1.customer class has subscriber class so it nested with one level customer--> subsciber. I think i'm not doing right: ...
AviC's user avatar
  • 111
2 votes
0 answers
821 views

Assigning parallel workers to predict from keras model using concurrent.futures

I am working on a Reinforcement learning project, where I have to gather a lot of data using a TensorFlow model. During the data gathering, the weights of the model do not change. So, I am using ...
kosa's user avatar
  • 121
2 votes
1 answer
52 views

Hardware driven data processing and ploting in need of better control flow

This is code for a measurement setup that receives a steady stream of UDP data, finds the trigger in one channel and operates on the data in the other channel to enhance the signal and remove noise, ...
Andreas Schuldei's user avatar
2 votes
0 answers
22 views

Implementation of Policy Gradient Reward Design paper

I've implemented the first experiment from the Reward Design via Online Gradient Ascent paper. I don't have any specific concerns, but it's my first time using multiprocessing or doing reinforcement ...
seed's user avatar
  • 121
-4 votes
1 answer
339 views

How can I reduce the TIme Complexity. of the python program

I have 7000 CSV files that I want to do operation on. but the problem is that it takes to much time like it takes 13 seconds to process these files. What I need to know is which operation take more ...
Hazrat ali's user avatar
3 votes
1 answer
5k views

python flask spawn jobs with multiprocessing

I have a python flask app that waits for requests from user app and than spawns a process with job based on the request it receives. It keeps the status and queue of the jobs in memory. The requests ...
Ondrej's user avatar
  • 133
5 votes
3 answers
170 views

Analysing a Huge Codebase with Python

I've written a Python program to analyse a huge C++ code base in excess of millions of lines of code. The job of the program is simply to search for all C and C++ style comments and extract all the ...
jess's user avatar
  • 1,378
3 votes
1 answer
138 views

Divide and Conquer Password Bruteforcer

My program brute-forces a password. The password is a string composed of a key and a four digit numeric code. The key is known so we are basically brute-forcing between 0000 through to 9999 An example ...
SamAko's user avatar
  • 247
4 votes
2 answers
415 views

Class with multiple workers

I am building a tool that interacts with a batched stream of incoming data. This data needs to be processed and the result returned. To split up the work I have created a class that has inbound (...
Alex's user avatar
  • 265
1 vote
1 answer
125 views

Parallel image segmentation [closed]

Follow up question Image processing using Python OOP library I want to use multiprocessing to analyze several images in parallel: ...
snowflake's user avatar
  • 275
-2 votes
1 answer
191 views

Python code refactor to use multiprocessing [closed]

I have written some Python code that opens files one by one, does some work, and writes data onto the filesystem. I figured, I want to make use of my cores.I've refactored the program into this. <...
snakeandrooster's user avatar
2 votes
1 answer
28 views

Join ArcGIS tables in Python of plant greenness in response to prior month's rainfall

I am using Python 3.8 with ArcGIS Pro 2.5. This code creates a joined table where I match values for plant greenness in the current month with rainfall X number of months into the past. Each loop ...
user5858's user avatar
  • 121
3 votes
1 answer
109 views

Data segmentation and optimization for Multi-threaded/Multi-core use (In Python)

I have written a very basic algorithm, that counts the amount of times sub-string appears in a string. The algorithm is given: seq - a string sequence k - length of a sub-string L - a window of a ...
rebelCoder's user avatar
3 votes
0 answers
78 views

Monitored parallel mapping across a Pandas dataframe

This code maps a function over a dataset in parallel in a monitored fashion, and returns the result in the variable out. ...
user5601's user avatar
  • 131
5 votes
1 answer
96 views

Create 1% Sample Using Multiprocessing in Python

I'm trying to process a large dataset (300GB, myfile.txt in the script) line by line using multiprocessing. I want to define a 1% random sample based one variable (contained in unique_ids_final.txt). ...
giacomo1488's user avatar
3 votes
1 answer
98 views

Plot timings for a range of inputs MkII

To celebrate questions getting more reputation points per upvote, I revisited my previous question, Plot timings for a range of inputs. Since the time of that original question (more than two years!),...
Graipher's user avatar
  • 41k
3 votes
2 answers
2k views

hash image url and if valid store image url

I am trying to get all image urls which are valid and unique.To make sure i wont end up using the same image again i calculate hash of the image and store it in a text file and read the text file ...
Ryan 's user avatar
  • 185
5 votes
1 answer
197 views

Gamblers ruin, parallel compute trials

Context The following script is a quick implementation of the gambler's ruin problem. For each given upper bound on the number of rounds in a game max_iter, ...
Quetzalcoatl's user avatar
3 votes
1 answer
56 views

Monitor a web page for when a desired price is met

I have a code that keeps checking a webpage with a price on it and when the price meets a number I set it to, the thing is purchased. I am looking for ways to increase the speed of it. I have added ...
Edwin Tan's user avatar
3 votes
1 answer
566 views

Simple sharding of delimited files to more sophisticated

A while back I came across the issue of having a large delimited file where I wanted to simply parallelize my python code across each line of the file. However, doing so all at once either took up too ...
SumNeuron's user avatar
  • 271
11 votes
2 answers
901 views

Parallelized for loop in Bash

I am using a Bash script to execute a Python script multiple times. In order to speed up the execution, I would like to execute these (independent) processes in parallel. The code below does so: <...
user213544's user avatar
1 vote
0 answers
980 views

First readers-writers problem using a single condition

For the first readers-writers problem, in which readers can access a given resource simultaneously, is the following Python3 solution correct? Problem parameters: One set of data is shared among a ...
Pedro Lopes's user avatar
2 votes
0 answers
1k views

Image processing using multiprocessing in Python

I have an image stack. I am trying to use multiprocessing to do some process on each image in stack to get new image, and then bitwise_or with the old image. Since ...
Hongbo Miao's user avatar
5 votes
0 answers
168 views

A Parallel Processing Template for Divide & Conquer Problems

I’ve written a program for solving a problem using standard single-threaded code. However, it looks like it could be recast as a multi-threaded problem using divide & conquer. Since this is a ...
davypough's user avatar
  • 405