Questions tagged [multiprocessing]
Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system.
-4
votes
0answers
20 views
Multiprocessing slowing script down [on hold]
I am writing a script to help me analyze and get statistics for poker hands. After a lot of research I was able to work with bit manipulation and get my statistics very efficiently for one hand. I ...
1
vote
0answers
28 views
Image processing using multiprocessing in Python
I have an image stack. I am trying to use multiprocessing to do some process on each image in stack to get new image, and then bitwise_or with the old image.
Since ...
-1
votes
0answers
31 views
Multiprocessing a for loop in Time series [closed]
I want to use multiprocessing for a 'For loop'. So basically from a another dataframe I am passing ID,PID to main dataframe, so depends on ID,PID it will filter the dataframe & build a model. I ...
4
votes
0answers
76 views
A Parallel Processing Template for Divide & Conquer Problems
I’ve written a program for solving a problem using standard single-threaded code. However, it looks like it could be recast as a multi-threaded problem using divide & conquer. Since this is a ...
10
votes
1answer
1k views
Python 3.7 UltimateBruteforcer
So I made a bruteforce program for sha-X hashes and MD5 using wordlists in Python.
I know Python is probably one of the worst languages for this, but it's only for learning purposes.
I think this was ...
1
vote
0answers
31 views
Python process-subprocess communication
I have two processes. First (main) one is "dealing" with user stuff. Second one is doing some internal stuff and should update its internal state according to user actions. So I've created two pipes. ...
14
votes
2answers
619 views
Translate nucleic acid sequence into its corresponding amino acid sequence
Goal of the program
The goal of the program is to translate a nucleic acid sequence into its corresponding amino acid sequence. The nucleic sequences
have to be formatted in a specific format called <...
5
votes
2answers
253 views
Implementing lock within a Python script
I've written a script in Python using multiprocessing to handle multiple process at the same time and make the scraping process faster. I've used locking within it to prevent two processes from ...
2
votes
2answers
108 views
Extract name, address and phone number from some web pages using multiprocessing
I've written a script in Python using the multiprocessing module to scrape values from web pages (one page per subprocess). As I'm very new to ...
3
votes
0answers
84 views
Using Perl's fork() to parallelize processing
I use bash scripts to process last names through a Perl program, and I parallelize the execution like this, by name initials:
...
3
votes
1answer
34 views
GET multiple data frames using external module's function that uses requests
Purpose
The point of this mini project is to quickly gather data from a website's API and combine the collected DataFrames into a "master" DataFrame with all of the stocks I am interested in looking ...
9
votes
2answers
264 views
Python worker pattern for multithreading
I'm building a script where I pass in a text file of hostnames, and a text file of commands and each command will then be run against each host.
To prevent this getting out of hand I'm also passing ...
2
votes
1answer
41 views
Count features in a file, using parallel code
When processing large files (say for parsing or performing computations based on the contents), we want to be able to use multiple processes to perform the job faster.
In my case I wanted to count ...
1
vote
1answer
928 views
Passing shared memory array to worker pool in Python
I'm working with big 2d numpy arrays, and a function that I need to iterate over each row of these arrays.
To speed things up, I've implemented parallel processing ...
3
votes
0answers
30 views
Architecture for a parsing tool
PostgreSQL, Python 3.5, SQLAlchemy
I have a main database which has a journals table containing 30k+ journals. For each journal I have a RSS feed URL, using which I can parse and get new articles. ...
4
votes
2answers
114 views
Parsing contents of a large zip file into a html parser into a .csv file
I have some zip files somewhere in the order of 2GB+ containing only html files. Each zip contains about 170,000 html files each.
My code reads the file without extracting them,
Passes the ...
2
votes
0answers
51 views
Python multiprocessing for tasks depending on each other
I have a simulation that runs a process on a tree structure, where the slow processes down processes are separate from each other. This looked like a thing that would be a prime candidate for ...
4
votes
0answers
98 views
Add an edge to a graph to minimize the average shortest path length
This program is used to find the nodes in a grid network, between which, if an edge is added, the average shortest path length of the entire grid reduces by the most.
"Average shortest path length" ...
4
votes
1answer
268 views
Genetic algorithm to find the minimum of a three-variable function
I've implemented a simple genetic algorithm for continuous floating point parameter spaces and without recombination. However, the way I'm passing in parameters and acquiring results from ...
0
votes
2answers
9k views
Loop through a list of URLs multi-threading and check return code in python
I have to loop through a list of over 4000 urls and check their http return code in python.
Url.txt: Contains a list of 4000 urls with one url per line.
The script takes a long time to run and I ...
4
votes
1answer
188 views
Subprocess CSV validator and multiprocessing on Python
This Python script takes a directory of CSV files and calls a Scala script which tests whether a file's contents match a given regular expression. The link to this Scala script can be found in the 3rd ...
4
votes
0answers
1k views
Python script to generate a playlist with the Spotify API
The result is just a list of youtube links.
I'm requesting an access token every time and throwing it away, so I'm thinking of a way to save it for future use. Maybe I'll use a tempfile, but even ...
4
votes
1answer
469 views
Creating a graph representing all combinations of 4-bit binary strings
Update: Important notes to understand the problem can be found in the comments under the response of colleague Gareth Rees's.
I have an algorithm that creates a graph that has all representations of ...
2
votes
1answer
244 views
Pipe based multiprocess filter
This code is a multi-threaded version of filter that uses pipes to communicate between the workers and the main thread. I am looking for any ways to make this code cleaner. I'm also interested in if ...
3
votes
0answers
102 views
Concurrency using immutability
I was watching the following video about software transactional memory(using a package that maintains an access log). At the moment I am trying to learn about concurrency with shared memory and ...
6
votes
3answers
211 views
Class scheduling tool with image output
I have a multilayered dictionary that contains information about classes. I am using this to code an automatic schedule builder that I will eventually add to a separate Tkinter application I made that ...
1
vote
0answers
637 views
Load balancing consumer and producer with multiprocessing
Tasks are distributed to producer in multiple worker process. Results from workers are sent through Queue to a ...
11
votes
1answer
223 views
Using Support Vector Machine algorithm to analyze satellite image
I am using Support Vector Machine (SVM) algorithm to perform a classification. The satellite image I am using is really big (5GB) that's why I am trying to take advantage of ...
1
vote
0answers
790 views
Watchdog for multiprocessing
The code shows here is a demo of my application. The func1 always hang somewhere with reason I haven't figured out by now. Now I just want to write a watchdog for this process. If it didn't generate ...
2
votes
0answers
563 views
Python multiprocessing with multithreading - processing a list of work
Although this code can be used for many different purposes, I'm building it for use in web-scraping. I would like to know if this is the best approach. Can you see any bugs or potential lockups?
The ...
4
votes
1answer
67 views
Simple Bash Parallel Tool (env_parallel dies on big env)
This is a simple tool to parallelize execution of multiple tasks in Bash, since env_parallel can't handle the environment size (or something of the sort - dies for ...
6
votes
2answers
11k views
Faster brute force algorithm
I have this brute force code, where you input a password and runs through combinations of numbers, lowercase and uppercase letters, and special characters until it match the password given.
The ...
6
votes
2answers
208 views
~600k external API requests for uploading part numbers to an inventory
I have a CSV containing ~600k part numbers to be uploaded to my website's inventory. However, this CSV only contains limited information. We're missing pricing and other related information. To get ...
14
votes
1answer
8k views
Python Executer that kills processes after a timeout
Imagine you want to run a bunch of tasks in parallel, but you can't be sure that all of your tasks will eventually terminate or rather decide to eat up all your memory and CPU time. This should be ...
3
votes
0answers
48 views
VIM colors downloader in Python, using multiprocessing
I recently posted this script:
VIM colors downloader in Python
But since I'm not allowed to update the code there, I wanted to get an idea on this version, that uses multiprocessing:
...
3
votes
1answer
359 views
Erlang process pool
I am coming from Java and I am trying to create a process pool in Erlang.
Like the Java API:
...
1
vote
0answers
73 views
Using RemoteChannels for an parallel iterable map
RemoteChannels hit with julia v0.5
There are no examples online, but I managed to get them to work.
They are still very new, and all this whole area of multithreading code continues to have running ...
6
votes
1answer
41 views
Fetch data from co-processor on network in background
My code communicates with a Raspberry Pi (with a static IP address) running on a network. The client sends a signal to the Pi, and the Pi sends back a double (encoded as a string) with the angle ...
5
votes
2answers
4k views
Multiprocess, multithreaded read write on a single file
I've got a job to create a library that would be able to support multithreaded and multi process read and write to a single file. On that single file we would store C# models in an array in JSON ...
2
votes
0answers
31 views
Adaptive scan of function concurrently
This function try to scan the given function in n dimensions in the given range by adaptively divide the area into n+1 points polygon (e.g. triangle in 2D; tetrahedron in 3D)
I tried to seperate the ...
1
vote
0answers
275 views
Optimal use of Python 3's multiprocessing package to create Sudoku grids
I want to create Sudoku grids using Python. As it's pretty slow if I want to create a bunch of grids, I decided to use multiprocessing.
My computer has 4 virtual ...
2
votes
2answers
1k views
Using doParallel to cycle through *.rds files
Overview
I have 165,000 *.rds files that are around 200,000 obs x 13 variables each. These files are unique to grid files (2.5 square mile data) so I need to keep them in their unique grid numbers. ...
1
vote
2answers
13k views
Blocking until condition is met
I have a class called _NodeProcess. It consists of a mainloop (main function) which is run in a separate process and methods (<...
3
votes
2answers
146 views
Asynchronous model fitting that allows termination in Python
The problem
When you work with Python interactively (e.g. in an IPython shell or notebook) and run a computationally intensive operation like fitting a machine-learning model that is implemented in a ...
1
vote
0answers
576 views
Simple distributed lock manager server based on zmq
I wrote a simple server to distribute locks on a network. It is used in a multi-server environment with a shared home directory to secure non-multiprocessing safe file writing (...
6
votes
1answer
137 views
Multiprocess Bogosort
I only program by hobby, and I'm trying to improve with Python, so along with practicing coding I decided to venture into multiprocessing. This is a command line and will sort a variable number of ...
7
votes
2answers
903 views
Extracting keywords from 3 billion CSV records
I am processing around 3 billion records using this piece of code. It's pretty slow; it would be really helpful if you could suggest better ways to do this.
...
7
votes
1answer
5k views
Fast Thread- and multiprocess-safe file operations in python under Linux
I'm trying to implement an alternative to python built-in open(), but safe for use in multi-threaded and multiprocessing environment. I'm using advisory locking, ...
8
votes
2answers
2k views
Mandelbrot fractal with MPI
I'm not very familiar with MPI. I've written this little piece of code to draw the Mandelbrot fractal processing each row of the image in parallel. It works but it's really slow. I've written code to ...
0
votes
3answers
159 views
Splitting many data files by quintile
The program below is about finding the values below the percentage and splitting them into five different output files. This works great for small input data files. But when I am trying to run it for ...
