Questions tagged [csv]
A comma-separated values (CSV) file stores tabular data (numbers and text) as plain text, with rows and columns delimited by line terminators and commas, respectively. You may also use this tag for variants where fields are separated by tabs or semicolons.
41 questions
16
votes
1
answer
39k
views
Efficiently filter a large (100gb+) csv file (v3)
I'm in the process of filtering some very(!) large files (100gb+): I can't download files with a lower granularity. This is a followup from this question.
The problem is as follows: I need to filter ...
- 698
8
votes
1
answer
12k
views
Printing out JSON data from Twitter as a CSV
I'm extremely new to Python, and the Twitter API, but I found an example online that walked me through the process. Now that I've been playing around with it for awhile, I've begun to push the limits ...
- 610
7
votes
3
answers
8k
views
Compute mean, variance and standard deviation of CSV number file
I've written my first C++ application this week and I wanted to see if there's anything that aren't up to standards or better ways to do what I'm currently doing.
For the goal of the exercise, let's ...
- 12.4k
6
votes
2
answers
2k
views
Efficiently filter a large (100gb+) csv file (v2)
EDIT: This question is followed up by this question.
I'm in the process of filtering some very(!) large files (100gb+): I can't download files with a lower granularity. This is a followup from this ...
- 698
2
votes
1
answer
963
views
Efficiently filter a large (100gb+) csv file (v1)
EDIT: This question was followed up by this question, with improved code.
I have a set of pretty large files (>2 GB, over 30m rows) containing intraday data in the following format:
...
- 698
20
votes
4
answers
10k
views
Web scraping the titles and descriptions of trending YouTube videos
This scrapes the titles and descriptions of trending YouTube videos and writes them to a CSV file. What improvements can I make?
...
- 1,080
12
votes
1
answer
1k
views
Find the peak stock price for each company from CSV data
During the hiring process, a company gave me this test:
Q) Consider Share prices for a N number of companies given for each month since year 1990 in a CSV file. Format of the file is as below with ...
- 223
10
votes
1
answer
55k
views
Writing lists to .csv file
I'm teaching myself programming, using Python as my initial weapon of choice.
I have learnt a few basics and decided to set myself the challenge of asking the user for a list of names, adding the ...
- 451
10
votes
2
answers
2k
views
Are there ways to speed up this string fuzzy matching in Golang?
I have a piece of python code doing fuzzy matching which works very well and is pretty fast. For reference, it uses the following files:
https://raw.githubusercontent.com/datasets/s-and-p-500-...
- 243
9
votes
3
answers
290
views
Collate votes on MP3 files to list them by popularity
For my first Python project, I decided to create a script that would automatically collate MP3 files into a music countdown. I wanted to 'learn by doing', so I dived straight in and used the Python ...
- 175
8
votes
3
answers
197
views
MP3 Vote Collator - Final
Here is the completed code for my first Python project, for which I received some help and feedback here: Collate votes on MP3 files to list them by popularity
Seeing as there have been considerable ...
- 175
8
votes
3
answers
601
views
Calculating effective shop inventories from CSV files
Program purpose
The purpose for the program is to create a file containing the effective inventory for each stocked item stored in a collection of shops.
Input and algorithm details
For clarity, ...
- 113
8
votes
1
answer
824
views
Outputting a .txt file from a .csv file
I wrote this little script in PHP a bit ago. It takes a .csv file and outputs a .txt file with certain data arranged in a "psuedo-array" looking format. I recently started messing around with Ruby so ...
- 183
7
votes
1
answer
3k
views
Efficiently adding huge amounts of data from CSV files into an SQLite DB in Java
I'm trying to parse values from a CSV file to a SQLite DB, however the file is quite large (~2,500,000 lines). I ran my program for a few hours, printing where it was up to, but by my calculation, the ...
- 191
7
votes
3
answers
616
views
A command line Java program for computing grade point average
(See the next iteration.)
I have this command line program for computing GPA (grade point average). Given credit sequence \$\langle c_1, c_2, \dots, c_n \rangle\$ and grade sequence \$\langle g_1, ...
- 29.9k
7
votes
4
answers
282
views
Blocking production lines in Python
I have the following code that works great, but takes 40 mins to 1 hours to finish the task. It has at least 30 rules for blocking lines, and 30k+ orders to check. I'm new to Python (3 months working ...
6
votes
1
answer
1k
views
CSV writer implementation
I've recently been assigned to write up a CSV writer with "as much flexibility" as possible concerning pretty much all (output stream, string fields with potential new line characters, separator ...
- 659
6
votes
1
answer
1k
views
Interface to a CSV dinosaur database
I have just finished with the first Ruby exercise on Level Up Rails, and wanted to get an idea on how I can refactor the code.
You can find the original exercise on github - including the ...
- 425
6
votes
1
answer
777
views
Read CSV and use bidirectional BFS to find the shortest connections between actors
I have a really huge file¹ with 17 million records in it.
Here is a sample of the file:
Actor Movie
1,2
2,2
3,1
4,3
2,3
...
6
votes
2
answers
3k
views
Calculate food company sales for the year
I am working on a Python project where a food processing company is trying to calculate its total sales for the year. Python has to read from a text file where its divided into for categories split by ...
- 61
5
votes
3
answers
12k
views
Creating a CSV stream from an object
I have this code that gets a CSV string from a list of objects where the first row in the string is the object's properties and the rows underneath are the properties values. I then create a ...
- 185
5
votes
1
answer
383
views
Batch upload users to nextcloud on docker with bash csv
I have this bash script that adds users to NextCloud from a csv file. It expects to be run from the same directory as the docker-compose.yml file.
I'm pretty new to bash scripting and I'm wondering ...
- 183
4
votes
2
answers
147
views
Dataset and frequency list generation by looping over files
I have been told that it would be wise to split up my code into semantically useful blocks. I tried the following but I need feedback.
What I already did:
As you can see I create two .csv files, so ...
- 495
4
votes
3
answers
309
views
A program to display, update and save a dictionary as .csv
I'm now looking for feedback on v2.0 of this program instead.
I'd like some feedback about my code. What bad habits do I have? What advice could help me write more Pythonic?
I'm trying to write a ...
4
votes
1
answer
2k
views
Converting JSON to CSV using powershell
I have a very large JSON Lines File with 4.000.000 Rows, and I need to convert several events from every row. The resulted CSV File contains 15.000.000 rows. How can I optimize this script?
I'm using ...
user223675
3
votes
2
answers
184
views
Validating a CSV list of contacts and convert it to JSON
I've written a class that takes a file, validates the formatting of the lines from an input file and writes the set of valid lines to an output file. Each line of the file should have a first name, ...
- 459
3
votes
2
answers
445
views
Java Reflection-based CSV Parser
I'm an experienced Java developer, however I don't have experience with reflection or Annotation classes. For fun, I tried to develop a CSV Reader class which can parse each row into a particular type....
- 267
3
votes
1
answer
222
views
A module to display, update and save a dictionary as .csv v2.0
I'd like feedback on my revised code after my previous question.
The goal is to create a module which can display, update, save dictionaries as .csv files. I'll use this in another program to update ...
3
votes
1
answer
438
views
Parsing positive integers from CSV file
I Needed to parse some positive integers with the following constraints:
If possible use C++ only (no C API)
Needs to be able to process parsed data after each newline
Should be reasonably fast
...
- 4,458
2
votes
5
answers
7k
views
CSV reader and parser
I have implemented a CSV reader. I think I did pretty well. Since CSV is a loosely defined format to begin with I decided to allow some malformations, like anything but a delimiter after an enclosed ...
- 242
2
votes
2
answers
985
views
Speeding up and fixing phone numbers from CSVs with Regex
I've hodgepodged together an attempt to extract all phone numbers from all CSVs in a directory, regardless of where they are and what format they're in. I want all phone numbers to be printed to a ...
- 391
2
votes
2
answers
6k
views
Creating a tuple from a CSV file
I have written code that reads in a CSV file and creates a tuple from all the items in a group. The group ID is in column 1 of the table and the item name is in column 2. The actual datafile is ~500M ...
- 85
2
votes
2
answers
1k
views
Comparing columns from two CSV files - follow-up
This is the second part of a question, you can find the first part here:
Comparing columns from two CSV files
I've made some changes to the script and here is what it looks like now:
...
- 367
2
votes
0
answers
133
views
Building nested object structures from CSV input
I'm reading big csv files and mashing the lines into hierarchical objects for analysis and display. The csv might look like this:
...
- 29
2
votes
1
answer
311
views
CSV writer implementation refactored
Based on my initial implementation found here and the suggested improvements in answers and comments, I've refactored the code.
The main changes are:
Removed redundant buffering.
Simplified ...
- 659
2
votes
2
answers
145
views
Receive data from SQL and CSV and send it to MySQL database using Python
Details:
I have a code, which does the following:
Get data from CSV and insert it into MySQL table
Get data from SQL Server and insert it into MySQL table
Run this code every 240 seconds - to ...
- 225
1
vote
1
answer
2k
views
Check that a data file contains the expected number of rows and columns of integers
This is a class I wrote check to check the format of .dat files. My code makes sure that the content of the files satisfies the conditions.
A file can be deemed invalid if:
The file can't be found.
...
- 37
1
vote
1
answer
157
views
Streaming learning OCaml
I wrote a simple online logistic regression, calibrated using gradient descent to compare the speed of an OCaml implementation vs the same Python script, executed with Pypy. It turned out that the ...
- 1,081
1
vote
1
answer
3k
views
Comparing columns from two CSV files
This question has a second part here:
Comparing columns from two CSV files - follow-up
I'm currently trying to compare two CSV files column by column but only when their indexes match.
Here is what ...
- 367
1
vote
2
answers
899
views
Student administration application using a .csv file for persistence
The application is a student administration system using a .csv file for persistence as part of our school project.
This is my first created a basic DAO. I'd like to have some feedback about what ...
user5498
-4
votes
3
answers
8k
views
Write object data to CSV files [closed]
Does this method follow the rules of objective programming? If not, how can I change it?
...
- 175