All Questions

Tagged with
Filter by
Sorted by
Tagged with
26
votes
5answers
3k views

Python web-scraper to download table of transistor counts from Wikipedia

I have been looking for answers for how to easily scrape data from Wikipedia into a CSV file with Beautiful Soup. This is the code so far. Is there an easier way to do it? ...
2
votes
1answer
54 views

Multithreading to process requests and save results in python

I was presented with a task to come up with a script that generates a CSV with POSTAL codes via bruteforce (I'm in Brazil, so that means CEP to us). Points to note: I'm using an external library, ...
2
votes
1answer
62 views

Remapping property IDs when inserting CSV data to a database table

My code is functioning the way I'm expecting, with the if. However, it is not pythonic to me, and that bugs me. The multiple if in the code below are supposed to ...
2
votes
1answer
65 views

Exporting data from PostgreSQL as CSV to S3 bucket

I have a general question regarding DRY. As you can see there are several parts such as io.StringIO(), csv.writer, ...
3
votes
0answers
54 views

JSON to CSV in python using json.loads and json_normalize

I am trying to convert a JSON file to CSV format using Python. I am using the JSON.loads() method and then using json_normalize()...
1
vote
1answer
22 views

Batch retrieve formatted address along with geometry (lat/long) and output to csv

I have a csv file with 3 fields, two of which are of my interest, Merchant_Name and City. My goal was to output multiple csv ...
2
votes
0answers
73 views

Pricing data checker script

I was recently given a task to implement a data-checker script which given an input file containing (date, last_price) values should check for 3 different kind of errors - missing values, stale values ...
3
votes
2answers
34 views

Excel To CSV Converter Ch 9 ATBS

I am learning more about python using the book "Automate The Boring Stuff". The problem I completed was "Excel To CSV Converter". I want to kick bad habits to the curb, and make sure I'm learning the ...
4
votes
0answers
32 views

Combining CSV files of simulation results

This script is a subset of a larger script where I have the output of many test simulations in the form of CSVs. Each file starts with the model name and includes the number of elements in my file. ...
2
votes
1answer
33 views

Grouping sales transactions by person

I have a csv file with sales transactions. Each transaction includes person identifiers (which are sometimes/often missing) and transaction data. Person identifiers are fname, lname, phone, email and ...
2
votes
0answers
26 views

Getting service rate based on time/day of service

I have a long roster where I'd like to categorize the rates by time/day of service. Most shifts ends the same date but there are some special cases where the shift ends on a different date. I have to ...
3
votes
3answers
129 views

Computing average duration per user from a large CSV file that doesn't fit in memory

We have a large log file which stores user interactions with an application. The entries in the log file follow the following schema: {userId, timestamp, actionType} where actionType is one of two ...
4
votes
1answer
35 views

Count words in a list of titles, with some cleanup

I have a list of article titles, where I wish to count the number of occurrences for each word (and remove some words and characters). The input is in a .csv file where the titles are in column '...
4
votes
0answers
46 views

Convert Amazon transaction data for import into accounting system

At the end of each billing cycle, Amazon generates a raw transaction file for my store's orders that cycle. I am converting that raw transaction file into a .csv file to be imported into my accounting ...
10
votes
2answers
1k views

Read/write a pipe-delimited file line by line with some simple text manipulation

This code that I wrote is supposed to read/write a pipe-delimited file line by line to a new file with some simple text manipulation. (It also adds two new columns) and publishes a "Status Update" ...
7
votes
1answer
183 views

Python code to identify structure of a text file

We have a department in our org that generates a lot of data in flat files, all with different formats. We are now trying to sort this data out and load into a database. As step 1, I am trying to ...
3
votes
1answer
70 views

Checking whether fund prices changed over multiple CSV files

A Mutual Fund data vendor is updating all Mutual Fund prices partially from 6 PM to early morning at 1 AM. Starting from 6 PM, I started to download prices for every 30 minutes. 1st download - 2 Fund ...
17
votes
4answers
1k views

Sensor logger for Raspberry Pi in a stratospheric probe

I'm writing a Python Script for a Raspberry Pi to measure different sensors. We are planning to send the Pi with that Script running to the stratosphere, so the power usage for the Pi is limited. I ...
3
votes
0answers
38 views

Generate CSV from multiple files on Regex Python Output

I have written a python script using regex to extrapolate a poorly formatted text file data table into an actual data table. I am currently printing the output using . C:\my file location\file.csv in ...
3
votes
1answer
72 views

CSV/JSON converter

This is my first Python script and I would like to hear the opinion of more experienced users, so I can do it right from the beginning instead of trying to correct my mistakes after months (years?) of ...
1
vote
1answer
64 views

Upload csv file to sqlserver with python

I have to weekly upload a csv file to sqlserver, and I do the job using python 3. The problem is that it takes too long for the file to be uploaded (around 30 minutes), and the table has 49000 rows ...
1
vote
0answers
28 views

Python Spreadsheet Header Indexer

At work, I find myself frequently crunching CSVs with headers prone to change names and location. I have a solution, but with no opportunity for code review I was hoping SE could offer a look ...
3
votes
1answer
49 views

Excel to CSV converter

To check out the cssv module i wrote a small programm which takes all xlsx files in a folder and converts them into csv files. Let me know what you would improve on this program. xlsx_tp_csv ...
2
votes
0answers
35 views

Speed up iteration for large dataset in python

I'm trying to parse data into a csv prior to a bulk transaction into Neo4j. I'm using vast amounts of data in the relationships and wondered if anyone could help in speeding up the transactions below....
7
votes
2answers
381 views

Grouping comma-separated lines together

I have comma-delimited files like these, where the first field is sorted in increasing order: Case 1 ( 1st file ) : ...
19
votes
4answers
5k views

Web scraping the titles and descriptions of trending YouTube videos

This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make? ...
2
votes
0answers
61 views

Detecting line separators and line breaks in Python CSV [closed]

Here is how I'm detecting a field separator and line break in a CSV file for which I do not already know the format. Does this look sufficient, or are there other things I should be adding here? <...
1
vote
1answer
84 views

Generate xyz coordinates of a tube made of multiple rings

In this code, I generate the coordinates of a tube made of multiple rings. Finally, I write those x,y,z coordinates in a .csv file. I want to simplify this code (maybe using a for loop?). ...
2
votes
2answers
612 views

Comparing columns from two CSV files - follow-up

This is the second part of a question, you can find the first part here: Comparing columns from two CSV files I've made some changes to the script and here is what it looks like now: ...
1
vote
1answer
1k views

Comparing columns from two CSV files

This question has a second part here: Comparing columns from two CSV files - follow-up I'm currently trying to compare two CSV files column by column but only when their indexes match. Here is what ...
0
votes
1answer
62 views

Taking text from a file and formatting it

My code takes numbers from a large text file, then splits it to organise the spacing and to place it into a 2-dimensional array. The code is used to get data for a job scheduler that I'm building. <...
-1
votes
1answer
402 views

Remove lines from csv if duplicate value in column [closed]

i am trying to write csv parser so if i have the same name in the name column i will delete the second name's line. For example: ...
2
votes
2answers
582 views

Split CSV file into a text file per row, with whitespace normalization

I have a CSV file that I would like to split up by row, do some whitespace processing on the string data in each row of the document column, and then output the processed data into separate TXT files ...
7
votes
2answers
490 views

Getting current weather data using OpenWeatherMap API

I would really appreciate if someone could take a look at this code. I am not very advanced, so ANY kind of feedback and critique would be precious to me. What this program is supposed to do is to go ...
2
votes
1answer
490 views

Hotel booking system help on how to book nights and memberships

I am doing a GCSE NEA project for practice. Customer Loyalty Scheme The managers of Crawdale Hotel Group have decided to update their customer loyalty scheme. Customers joining the scheme ...
4
votes
0answers
1k views

Python CSV to XML converter

I'm creating an application that reads in data from a CSV file and creates an XML file using LXML. The below code works as expected. However, before I develop it further I would like to refactor it ...
1
vote
1answer
58 views

CSV file cell slicer

This program is supposed to find and slice any cell within a CSV file which contains more than a defined number of characters. The files can be pretty large, so being a beginner I would like to know ...
4
votes
2answers
236 views

Calculating travel statistics based on CSV data

I took a code test for acme transport company and was rejected. That's ok. I'm still learning how to code. I didn't get much feedback so that's a bit frustrating. Can anyone here provide tips on how ...
6
votes
1answer
126 views

Personal School Course Scheduler

A good friend of mine had the challenge of trying to build a schedule using all available times and classes through a spreadsheet... by hand. He asked me if I could build a program to generate valid ...
4
votes
0answers
2k views

Fastest way to write large CSV file in python

I'm fairly new to python and pandas but trying to get better with it for parsing and processing large data files. I'm currently working on a project that requires me to parse a a few hundred CSV CAN ...
4
votes
2answers
166 views

Iterate over a list of list names as file names

Interrogating a web API using query-url's, for each query I can get either zero hits, one hit, or multiple hits. Each of those categories needs to go into a separate CSV file for manual review or ...
6
votes
2answers
2k views

Split big file into separate files depending on column's distinct values

I have a big file, which contains column names in the first row. The data is about 20GB and consists of 10s of millions of rows. Every other row contains data which I would like to append to other ...
8
votes
2answers
56 views

Streaming scheduled TV shows according to a CSV playlist

I'm very new to Python, and I've made a small program/script that will allow me to stream specific files at a given time. My idea is that I have a .csv file for each day, with one row for each movie/...
1
vote
0answers
35 views

Grouped by First Two Digits of Account_ID of ~240 million rows of Pandas DF

I have 40 CSV files, each containing ~6 million rows, and 2 columns. The columns are account_id and account_balance. The ...
1
vote
1answer
504 views

Handling periods ('.') in CSV column names with Pandas

I'm reading and processing a fairly large csv using Pandas and Python 3.7. Header names in the CSV have periods in them ('full stops', Britons say). That's a problem when you want to address data ...
4
votes
2answers
139 views

Parsing contents of a large zip file into a html parser into a .csv file

I have some zip files somewhere in the order of 2GB+ containing only html files. Each zip contains about 170,000 html files each. My code reads the file without extracting them, Passes the ...
11
votes
3answers
836 views

Aggregate prescriptions per drug in a CSV file

I have a question that is similar in spirit to this previously asked question. Nonetheless, I can't seem to figure out a suitable solution. Input: I have CSV data that looks like (FICTIONAL ...
1
vote
1answer
207 views

Scraping content from yellowpage and write them to a csv file

I've written a script in python to scrape names and phone numbers of a certain search from yellowpage traversing multiple pages. The scraper has the ability to collect all the links connected to all ...
2
votes
1answer
95 views

Convenience function for writing files in CSV, JSON, etc

The McCabe code complexity of the following function is 19 as found by pytest --mccabe test.py (from my toy-project mpu). While ...
5
votes
3answers
4k views

Writing to a csv file in a customized way using scrapy

I've written a script in scrapy to grab different names and links from different pages of a website and write those parsed items ...