Code Review

Questions tagged [multiprocessing]

Ask Question

Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system.

74 questions
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
2
votes
0answers
12 views

Assigning parallel workers to predict from keras model using concurrent.futures

I am working on a Reinforcement learning project, where I have to gather a lot of data using a TensorFlow model. During the data gathering, the weights of the model do not change. So, I am using ...
2
votes
1answer
41 views

Hardware driven data processing and ploting in need of better control flow

This is code for a measurement setup that receives a steady stream of UDP data, finds the trigger in one channel and operates on the data in the other channel to enhance the signal and remove noise, ...
2
votes
0answers
15 views

Implementation of Policy Gradient Reward Design paper

I've implemented the first experiment from the Reward Design via Online Gradient Ascent paper. I don't have any specific concerns, but it's my first time using multiprocessing or doing reinforcement ...
-4
votes
1answer
52 views

How can I reduce the TIme Complexity. of the python program

I have 7000 CSV files that I want to do operation on. but the problem is that it takes to much time like it takes 13 seconds to process these files. What I need to know is which operation take more ...
3
votes
1answer
191 views

python flask spawn jobs with multiprocessing

I have a python flask app that waits for requests from user app and than spawns a process with job based on the request it receives. It keeps the status and queue of the jobs in memory. The requests ...
6
votes
3answers
108 views

Analysing a Huge Codebase with Python

I've written a Python program to analyse a huge C++ code base in excess of millions of lines of code. The job of the program is simply to search for all C and C++ style comments and extract all the ...
3
votes
1answer
64 views

Divide and Conquer Password Bruteforcer

My program brute-forces a password. The password is a string composed of a key and a four digit numeric code. The key is known so we are basically brute-forcing between 0000 through to 9999 An example ...
4
votes
2answers
159 views

Class with multiple workers

I am building a tool that interacts with a batched stream of incoming data. This data needs to be processed and the result returned. To split up the work I have created a class that has inbound (...
1
vote
1answer
69 views

Parallel image segmentation [closed]

Follow up question Image processing using Python OOP library I want to use multiprocessing to analyze several images in parallel: ...
-2
votes
1answer
42 views

Python code refactor to use multiprocessing [closed]

I have written some Python code that opens files one by one, does some work, and writes data onto the filesystem. I figured, I want to make use of my cores.I've refactored the program into this. <...
2
votes
1answer
24 views

Join ArcGIS tables in Python of plant greenness in response to prior month's rainfall

I am using Python 3.8 with ArcGIS Pro 2.5. This code creates a joined table where I match values for plant greenness in the current month with rainfall X number of months into the past. Each loop ...
4
votes
1answer
61 views

Data segmentation and optimization for Multi-threaded/Multi-core use (In Python)

I have written a very basic algorithm, that counts the amount of times sub-string appears in a string. The algorithm is given: seq - a string sequence k - length of a sub-string L - a window of a ...
4
votes
0answers
30 views

Monitored parallel mapping across a Pandas dataframe

This code maps a function over a dataset in parallel in a monitored fashion, and returns the result in the variable out. ...
5
votes
1answer
81 views

Create 1% Sample Using Multiprocessing in Python

I'm trying to process a large dataset (300GB, myfile.txt in the script) line by line using multiprocessing. I want to define a 1% random sample based one variable (contained in unique_ids_final.txt). ...
3
votes
1answer
56 views

Plot timings for a range of inputs MkII

To celebrate questions getting more reputation points per upvote, I revisited my previous question, Plot timings for a range of inputs. Since the time of that original question (more than two years!),...
2
votes
2answers
532 views

hash image url and if valid store image url

I am trying to get all image urls which are valid and unique.To make sure i wont end up using the same image again i calculate hash of the image and store it in a text file and read the text file ...
5
votes
1answer
118 views

Gamblers ruin, parallel compute trials

Context The following script is a quick implementation of the gambler's ruin problem. For each given upper bound on the number of rounds in a game max_iter, ...
3
votes
1answer
47 views

Monitor a web page for when a desired price is met

I have a code that keeps checking a webpage with a price on it and when the price meets a number I set it to, the thing is purchased. I am looking for ways to increase the speed of it. I have added ...
3
votes
1answer
169 views

Simple sharding of delimited files to more sophisticated

A while back I came across the issue of having a large delimited file where I wanted to simply parallelize my python code across each line of the file. However, doing so all at once either took up too ...
11
votes
2answers
576 views

Parallelized for loop in Bash

I am using a Bash script to execute a Python script multiple times. In order to speed up the execution, I would like to execute these (independent) processes in parallel. The code below does so: <...
1
vote
0answers
459 views

First readers-writers problem using a single condition

For the first readers-writers problem, in which readers can access a given resource simultaneously, is the following Python3 solution correct? Problem parameters: One set of data is shared among a ...
1
vote
0answers
803 views

Image processing using multiprocessing in Python

I have an image stack. I am trying to use multiprocessing to do some process on each image in stack to get new image, and then bitwise_or with the old image. Since ...
5
votes
0answers
124 views

A Parallel Processing Template for Divide & Conquer Problems

I’ve written a program for solving a problem using standard single-threaded code. However, it looks like it could be recast as a multi-threaded problem using divide & conquer. Since this is a ...
10
votes
1answer
1k views

Python 3.7 UltimateBruteforcer

So I made a bruteforce program for sha-X hashes and MD5 using wordlists in Python. I know Python is probably one of the worst languages for this, but it's only for learning purposes. I think this was ...
1
vote
0answers
44 views

Python process-subprocess communication

I have two processes. First (main) one is "dealing" with user stuff. Second one is doing some internal stuff and should update its internal state according to user actions. So I've created two pipes. ...
14
votes
2answers
680 views

Translate nucleic acid sequence into its corresponding amino acid sequence

Goal of the program The goal of the program is to translate a nucleic acid sequence into its corresponding amino acid sequence. The nucleic sequences have to be formatted in a specific format called <...
5
votes
2answers
273 views

Implementing lock within a Python script

I've written a script in Python using multiprocessing to handle multiple process at the same time and make the scraping process faster. I've used locking within it to prevent two processes from ...
2
votes
2answers
757 views

Extract name, address and phone number from some web pages using multiprocessing

I've written a script in Python using the multiprocessing module to scrape values from web pages (one page per subprocess). As I'm very new to ...
3
votes
0answers
247 views

Using Perl's fork() to parallelize processing

I use bash scripts to process last names through a Perl program, and I parallelize the execution like this, by name initials: ...
3
votes
1answer
59 views

GET multiple data frames using external module's function that uses requests

Purpose The point of this mini project is to quickly gather data from a website's API and combine the collected DataFrames into a "master" DataFrame with all of the stocks I am interested in looking ...
9
votes
2answers
3k views

Python worker pattern for multithreading

I'm building a script where I pass in a text file of hostnames, and a text file of commands and each command will then be run against each host. To prevent this getting out of hand I'm also passing ...
2
votes
1answer
64 views

Count features in a file, using parallel code

When processing large files (say for parsing or performing computations based on the contents), we want to be able to use multiple processes to perform the job faster. In my case I wanted to count ...
2
votes
1answer
4k views

Passing shared memory array to worker pool in Python

I'm working with big 2d numpy arrays, and a function that I need to iterate over each row of these arrays. To speed things up, I've implemented parallel processing ...
3
votes
0answers
31 views

Architecture for a parsing tool

PostgreSQL, Python 3.5, SQLAlchemy I have a main database which has a journals table containing 30k+ journals. For each journal I have a RSS feed URL, using which I can parse and get new articles. ...
4
votes
2answers
216 views

Parsing contents of a large zip file into a html parser into a .csv file

I have some zip files somewhere in the order of 2GB+ containing only html files. Each zip contains about 170,000 html files each. My code reads the file without extracting them, Passes the ...
2
votes
0answers
169 views

Python multiprocessing for tasks depending on each other

I have a simulation that runs a process on a tree structure, where the slow processes down processes are separate from each other. This looked like a thing that would be a prime candidate for ...
6
votes
1answer
208 views

Add an edge to a graph to minimize the average shortest path length

This program is used to find the nodes in a grid network, between which, if an edge is added, the average shortest path length of the entire grid reduces by the most. "Average shortest path length" ...
3
votes
1answer
381 views

Genetic algorithm to find the minimum of a three-variable function

I've implemented a simple genetic algorithm for continuous floating point parameter spaces and without recombination. However, the way I'm passing in parameters and acquiring results from ...
1
vote
2answers
17k views

Loop through a list of URLs multi-threading and check return code in python

I have to loop through a list of over 4000 urls and check their http return code in python. Url.txt: Contains a list of 4000 urls with one url per line. The script takes a long time to run and I ...
4
votes
1answer
388 views

Subprocess CSV validator and multiprocessing on Python

This Python script takes a directory of CSV files and calls a Scala script which tests whether a file's contents match a given regular expression. The link to this Scala script can be found in the 3rd ...
5
votes
0answers
2k views

Python script to generate a playlist with the Spotify API

The result is just a list of youtube links. I'm requesting an access token every time and throwing it away, so I'm thinking of a way to save it for future use. Maybe I'll use a tempfile, but even ...
4
votes
1answer
600 views

Creating a graph representing all combinations of 4-bit binary strings

Update: Important notes to understand the problem can be found in the comments under the response of colleague Gareth Rees's. I have an algorithm that creates a graph that has all representations of ...
2
votes
1answer
339 views

Pipe based multiprocess filter

This code is a multi-threaded version of filter that uses pipes to communicate between the workers and the main thread. I am looking for any ways to make this code cleaner. I'm also interested in if ...
3
votes
0answers
109 views

Concurrency using immutability

I was watching the following video about software transactional memory(using a package that maintains an access log). At the moment I am trying to learn about concurrency with shared memory and ...
6
votes
3answers
235 views

Class scheduling tool with image output

I have a multilayered dictionary that contains information about classes. I am using this to code an automatic schedule builder that I will eventually add to a separate Tkinter application I made that ...
1
vote
0answers
998 views

Load balancing consumer and producer with multiprocessing

Tasks are distributed to producer in multiple worker process. Results from workers are sent through Queue to a ...
11
votes
1answer
276 views

Using Support Vector Machine algorithm to analyze satellite image [closed]

I am using Support Vector Machine (SVM) algorithm to perform a classification. The satellite image I am using is really big (5GB) that's why I am trying to take advantage of ...
1
vote
0answers
1k views

Watchdog for multiprocessing

The code shows here is a demo of my application. The func1 always hang somewhere with reason I haven't figured out by now. Now I just want to write a watchdog for this process. If it didn't generate ...
2
votes
0answers
632 views

Python multiprocessing with multithreading - processing a list of work

Although this code can be used for many different purposes, I'm building it for use in web-scraping. I would like to know if this is the best approach. Can you see any bugs or potential lockups? The ...
4
votes
1answer
88 views

Simple Bash Parallel Tool (env_parallel dies on big env)

This is a simple tool to parallelize execution of multiple tasks in Bash, since env_parallel can't handle the environment size (or something of the sort - dies for ...

1
2