Questions tagged [csv]

A comma-separated values (CSV) file stores tabular data (numbers and text) as plain text, with rows and columns delimited by line terminators and commas, respectively. You may also use this tag for variants where fields are separated by tabs or semicolons.

Filter by
Sorted by
Tagged with
7
votes
1answer
6k views

CSV file reader in PHP that supports large files (>15k lines)

I have written the following function in PHP to read a CSV file. It works correctly for small files. However, if I try to read in files that are bigger than 15k lines, it takes between 1–2 ...
3
votes
2answers
75 views

Sort a file on the first two columns, with optional filtering

I am a newbie to Rust and low-level coding. I come from data science and I have always used scripting languages so I may have done weird or useless things (with lifetimes and references). This is ...
1
vote
1answer
44 views

script that imports only data that is not in the database yet

I want to write a script that imports data into a database (in django). The database has a Company class with a name (CharField) instance. Additionally, it has a Person class with name (CharField), ...
4
votes
1answer
123 views

Data comparing and difference calculating python program

The following is a python program processing files input and generating a .csv output. There are two input files of .txt format, ...
1
vote
1answer
100 views

Read in a file, check that it meets certain criteria

This was my original posting: Check that a data file contains the expected number of rows and columns of integers I updated the code. Still what could I improve on? Another question I have is how ...
1
vote
1answer
2k views

Check that a data file contains the expected number of rows and columns of integers

This is a class I wrote check to check the format of .dat files. My code makes sure that the content of the files satisfies the conditions. A file can be deemed invalid if: The file can't be found. ...
2
votes
1answer
57 views

Retrieving records from a CSV file with a geographical hierarchy

I have the following scenario and I would like some review or suggestions. I have a CSV file with lot of records. The records are basically as follows: There are names of counties in Kenya. These ...
1
vote
1answer
823 views

Extracting certain products from a webpage using Scrapy

I've written a script in Python Scrapy to harvest various product names and price from books.toscrape. The reason I submit this tiny code to Code Review is because, in Python 3 when it comes to work ...
2
votes
1answer
204 views

ETL project, handling XML, CSV, and text files

I have completed an ETL project to collect, parse and load files. I decided to make it clean OOP way using interfaces and abstract, but have some questions below. ...
6
votes
1answer
3k views

Count occurrences of date-time in large CSV

My input file is a CSV of varying size, up to around 10GB. The file has several fields, however I'm only interested in the third column, a date-time field. The date-times are in UTC timezone, and they ...
5
votes
1answer
90 views

Saving web search results for EANs to CSV

This is my first program in Python. Following code is taken from different posts (mostly this site) and combined together so that I can automate my routine task. It's working properly but I need ...
2
votes
1answer
36 views

linux rearange field and sort by column

I have (tab-separated) input files: id1 id2 .... id1 id3 .... id3 id4 .... id2 id1 .... id3 id4 .... ..... I need to ...
1
vote
2answers
6k views

Opening a CSV file in Python, with error handling

Trying to improve my function, as will be used by most of my code. I'm handling most common exception (IOError) and handling when data has no values. ...
2
votes
2answers
145 views

Finding football team with the smallest difference in goals

The file football.dat contains the results from the English Premier League for 2001/2. The columns labeled ‘F’ and ‘A’ contain the total number of goals scored for and against each team in that season ...
5
votes
3answers
930 views

WV lottery tool in Python

I am self taught in python. I love to play the lottery and play with numbers. I put this program together as a hobby to learn python and build something that people can use. I used and reused code ...
2
votes
1answer
153 views

Adding values to rows in CSV

This code below is for one of my scripts that I am trying to automate. I am dealing with the rows and all the data that is in those rows. What I am trying to do is make this look neater and maybe ...
5
votes
1answer
55 views

Aligning fstab columns

I know I can use the unix column function to replicate 90% of the functionality here, so this is mostly just to practice python. I find a misaligned fstab file really hard to read. This script parses ...
3
votes
2answers
25 views

Efficient way of reading in list of string csvs' into lists

I have a list of strings. Each string is a comma separated list of numbers. I want to quickly convert each of these strings into a list. So far I have done the following: My Solution So Far ...
2
votes
1answer
91 views

The fastest way to import and edit data in R

So I want to import and edit a huge amount of data. I have a file with multiple data sets from time series that are chronologically ordered. Now I want to open them and edit to have a final dataset ...
1
vote
1answer
6k views

Split data from single CSV file into several CSV files by column value

Okay...so I have been self teaching for about seven months, this past week someone in accounting at work said that it'd be nice if she could get reports split up....so I made that my first program ...
3
votes
1answer
1k views

Download and write ~100k images

Currently I will be downloading around ~100K images. This process may run on a weekly basis, I'm considering the following factors: HTTP Requests (Will be taken care by external LB, so same source IP ...
5
votes
0answers
559 views

Converting multiple excel files in multiple folders into CSV

The whole project is quite large, involving parsing the CSV-files, validating them, exporting them to a database and such, but my code in question involves subset of the project: converting multiple ...
1
vote
1answer
179 views

Crawler for performing reverse search and writing results into a CSV file

I've written a script which is able to perform reverse search in the website using the Name and Lid from a predefined CSV file. However, when the search is done then it can put the results containing ...
3
votes
0answers
171 views

Scrape a stock value and add it to a csv file along with other data. [Python 3]

Any and all suggestions are welcome, mainly looking for suggestions on clean up, and making it more readable as I feel that it is inefficient and not very clean. Some specific things: A better way ...
1
vote
1answer
551 views

Output a flight schedule, formatted in military time

I have a CVRP problem that I solved in Python. There are 6 planes (T1-T6) and 3 airports (AUS,DAL,HOU). The airports have 1,2,3 gates respectively. Each airport has a minimum ground time of 25, 30, 35 ...
5
votes
3answers
278 views

CSV parser for a tech test

The requirements: The following was an interview test for a junior position. You’ve been asked to write a program that will take a dictionary file (csv list of words) and output the number of ...
2
votes
0answers
182 views

Split CSV into Seperate CSV Files on Parent Row (based on segment count)

I have a csv that is exported from SolidWorks, the input looks like this: ...
4
votes
2answers
616 views

Python Script: Converts .tsv to TFRecord

The python code below converts a tab spaced values file (.tsv) into a TFRecord in hopes that this approach expedites future time spent loading the data. Is the performance of this task correct and ...
3
votes
0answers
6k views

Powershell script that reads Outlook emails and exports results as .csv

I wrote my first powershell script. It reads through all the emails in a folder and exports some of the data as a csv file. I'm happy with the way it works but I plan to share it with co-workers, so ...
3
votes
1answer
60 views

Filtering and processing computers listed in a CSV file

I am trying to make my script run a lot faster. I have a csv that has about 330k of lines on it that I plan on running my script with and it has been taking an extremely long time. I just did ran the ...
5
votes
4answers
98 views

Finding students with the best grades in a given subject

We have a school assignment, in which the program is to read data of students from stdin, in the following format: LASTNAME FIRSTNAME STUDENT_ID SUBJECT AVERAGE_GRADES. The program has to print out ...
2
votes
1answer
173 views

Read CSV files and find the same values on the specific columns

I created a simple command line application to read CSV files in a specific folder and find the same values on the specify column which exists on all files. (Here's the complete source code.) Here's ...
10
votes
2answers
424 views

In memory data access for CSV files that may have changed

I'm reading data from CSV files and store the result in a simple DataContext instance that gets injected throughout my application. ...
4
votes
1answer
3k views

Mapping a CSV with UTF-8 encoding to JSON in Python

I'm not a programmer, just a guy who needs to solve particular things using code. I needed to create a JSON from a CSV with predefined headers. I'm downloading an attachment from JIRA (REST API), ...
3
votes
1answer
124 views

CSV Splitter in Bash

I am splitting a csv file where the first 3 columns will be common for all the output files. input file: ...
0
votes
1answer
46 views

Generates a CSV output of dummy data

I have written a quick piece of Java code to create some CSV data. The code simply outputs all of the permutations of specified lists of values and gives each row a unique value. I want to know if ...
7
votes
2answers
1k views

Append blank value(s) to missing fields in a CSV line

I have a small app that takes in CSV files. It reads the CSV file and before it inserts a line into the DB it checks certain conditions on the line. Now, let's say the length of the line should be 5. ...
4
votes
1answer
2k views

Create SQL statements from CSV file for creating a MySQL table

This program reads in a CSV file's header row and writes out SQL statements to a .sql file that can be used to create a new table in a MySQL database and load the data into it. If anyone can offer ...
2
votes
1answer
65 views

Many CSVs to list, optimize

I have a Python3 script (running on Python 3.5.2) which reads 110 CSV files (tab delimited) into a list. The largest file is 20 MB and the list ends up looking like this: ...
3
votes
2answers
5k views

Mapping .csv rows to a Dictionary

I have a .csv file in which each row represents an association between a software application and a web server. My goal is to fetch a List<Server> by ...
2
votes
2answers
360 views

Loading CSV into MySQL using OOP PHP

I need to make a simple CSV->MySQL script in OOP and I was wondering if there's anything I am doing wrong in a object-oriented sense. I have a file name db.php where I connect to MySQL via MySQLi and ...
3
votes
2answers
588 views

Read numeric data from a text file using C++

I need to read Numeric Data from a text file that looks like this: 2 cell X-cord Y-cord 1 1.1 2.2 2 2.1 2.0 The first line indicates the dimension (...
3
votes
2answers
79 views

Extracting data from database to CSV

I have a feature for users to be able to export the database information. But if the user choose all options to export it takes 1 minute or more to download the .csv file. I'm only including 1 part of ...
5
votes
2answers
198 views

Validating a CSV list of contacts and convert it to JSON - Follow Up

This is a follow up to my original post: I've written a class that takes a file, validates the formatting of the lines from an input file and writes the set of valid lines to an output file. Each ...
3
votes
1answer
61 views

Python command-line program to convert genomic data file

Background: I have written this code to transforme a .csv file exported from a software called Geneious containing SNPs and concatenate them into a DNA sequence. So basically take fields from .csv ...
8
votes
4answers
8k views

Convert XML to CSV

I'm pretty sure this code can be optimized, but I'm not talented enough in Linq to do it myself. Here's what I'm trying to do: I have an XML file that needs to be converted into a .csv file. The XML ...
3
votes
2answers
161 views

Validating a CSV list of contacts and convert it to JSON

I've written a class that takes a file, validates the formatting of the lines from an input file and writes the set of valid lines to an output file. Each line of the file should have a first name, ...
1
vote
1answer
90 views

Setting variables from a text file

I'm new to PHP, having coded in Perl and ColdFusion years ago. In the process of trying to familiarize myself with my new job, I have found some code that would neither seem efficient nor logical, ...
7
votes
1answer
3k views

Football game simulation

I'm working on a text-based football simulation game along the lines of Football Simulator. Below is a subset of my total code, specifically the functions used to create a new player. I also have ...
6
votes
2answers
1k views

Python code to split csv into smaller csvs, not splitting IDs

I have Python code that splits a given large csv into smaller csvs. This large CSV has an ID column (column 1), which consecutive entries in the csv can share. The large csv might look something like ...