There are a few performance tricks we can apply here:
add __slots__ to the class definition should help with memory and performance as well:
"""Set up regular expressions"""
# use https://regexper.com to visualise these if required
_reg_school = re.compile(r'School = (.*)\n')
_reg_grade = re.compile(r'Grade = (.*)\n')
I would try to use existing pandas features where possible to keep this code minimal - this aids readability and reduces the possibility of bugs being introduced in complicated loop structures.
from scipy.stats import chi2_contingency
def chisq_of_df_cols(df, c1, c2):
groupsizes = df.groupby([c1, c2]).size()
ctsum = groupsizes.unstack(...
I think you need create boolean DataFrame by compare all filtered columns values by scalar for not equality and then check all Trues per rows by all:
df = df[(df[['A','C']] != 0).all(axis=1)]
A B C
0 1 2 5
2 6 8 4
print (df[['A','C']] != 0)
0 True True
1 True False
2 True True
3 False True
Let's start with the obvious: this code doesn't run. You're missing ans = starter() so that further (el)if ans.lower() == ... doesn't miserably fail with a NameError.
Likely, you define q() but never use it.
And you also appears to have other useless stuff floating around: why use both textblob and indicoio to perform sentiment analysis? You also seem to ...
I always get my code to be as clean as possible before starting work on performance. Here you have a bare except, which can hide errors. Change it to something better, for now I'll change it to Exception. You are also hiding a potential IndexError and AttributeError if the data ever changes shape as they are in the try.
You’re using the wrong tool for the job. Basically, you do all the computation in Python, use numpy for intermediate storage and pandas for display.
Instead, you should compute the list of tribonacci numbers and from there on use pandas for anything else as it would be much more efficient / readable. I’d keep building the tribonacci numbers in Python as I ...
Using crosstab, this can be done in a single step:
It will give this desired result:
var1 | 0 1
0 |0 1
1 |2 0
2 |2 0
You can name the index and colnames and also get the row totals and column totals:
I've never used numpy or matplotlib, so I can only speak to issues of style.
You're allowing for far too much nesting here. Your code consists of a giant, dense, deeply nested chunk. As a result, the eyes have very few good landmarks to rest on; making your code hard to read.
Put your existing function definitions at the very top level outside of any ...
AFAIK it does what you want (but on this site you should generally be
sure that your code does what you want beforehand).
Beauty, eye of the
...; that said, this code can be rewritten in a very concise manner:
import pandas as pd
import scipy.stats as scs
return range(int(series.min()), int(series.max()) + 1)
It doesn't look like you really need regular expressions. This construct just using basic string operations is about 10x faster than the construct with the regular expressions:
present_colors = set()
for value in ex_df['colors'].values:
for color in [x.strip() for x in value.split(';')]:
And a bit ...
Iterating in Python is slow, iterating in C is fast. So, it's best to keep as much as possible within Pandas to take advantage of its C implementation and avoid Python.
You can use apply on groupby objects to apply a function over every group in Pandas instead of iterating over them individually in Python.
df["metric1_ewm"] = df.groupby("person").apply(...
Use the diff().
x['time_delta'] = x.timestamp.diff().fillna(x['time_delta'])
This works as below, in a simpler example.
You could use the diff() Series method (with fillna to replace the first value in the series):
s = pd.Series([11, 13, 56, 60, 65])
This was compiled from ...
I'm going to put this code into an editor, "proofread" it from top to bottom, give notes as I go, and then paste the final result to show the effect of the edits.
Code editors don't usually wrap lines since linebreaks mean things in code. Consequently, most coding style guidelines suggest limiting your column width so the reader doesn't need to scroll ...
You have eight conditions to match for every UPDATE. A typical solution would store timestamps using a DATETIME or TIMESTAMP column, so that there is only one value to match.
For reasonable performance, ensure that the timestamp field is indexed.
When opening very large files, first concern would be memory availability on your system to avoid swap on slower devices (i.e. disk).
Pandas is shipped with built-in reader methods. For example the pandas.read_table method seems to be a good way to read (also in chunks) a tabular data file.
In the specific case:
df = pandas.read_table('./...
As you already use pandas, you are right that there is no overhead on using Timestamps (the kind of objects returned by pd.to_datetime). But, as said in the documentation (help(pd.Timestamp)):
Help on class Timestamp in module pandas.tslib:
| TimeStamp is the pandas equivalent of python's Datetime
| and is ...
You take the combinations in a way too convoluted way. For starter, I would simplify the retrieval of "same memberid" questions:
for memberid in IncorrectQuestions.memberid.unique():
questions = IncorrectQuestions[IncorrectQuestions.memberid == memberid].questionid
No need to groupby here since you’re already filtering by the one memberid you’re ...
I am the Adams of Adams-Skopek. I want to point out a couple of things:
You get some truncation error when you sum or multiply a series of values in different orders. You need to estimate this and account for it.
There is a lower limit on sparsity of the data, you need to address this.
If you can get a copy of the Pagano and Halverson program (cited in ...
DISCLAIMER: I don't know cython and have never used it, so if any of my advice doesn't apply because of cython limitations, feel free to disregard it.
Counting neighbors in a game board is very easy to do via convolution with the proper kernel. Below, I used SciPy's convolve function, not the much faster fftconvolve, because the latter only works on floats ...
Setting aside PEP 8 (official style guide) issues I would make the following change:
Rather than keeping your politician bins, names and email addresses in separate data structures (the only way they are tied right now is by the array index number) - I would create a Politician class. Something like:
def __init__(self, name, ...
Imagine how much of a pain it would be to add another politician to this list. In addition to the Politician class suggested in another answer, I would suggest creating a list of Politicians and whenever you want to do something to one of them, iterate over this list. This may require adding a method to your Politician class. Then, adding another candidate ...
nlargest could help, it finds the maximum n values in pandas series.
In this line of code, groupby groups the frame according to state name, then apply finds the 3 largest values in column CENSUS2010POP and sums them up. The resulting series has the unique state names as index and the corresponding top 3 counties sum, applying nlargest gets a series with ...
SUMLEV is explained here
Definitely want to use nlargest
The advantage of nlargest is that it performs a partial sort therefore in linear time.
However, you don't want to groupby twice. So we'll define a helper function in which we only groupby once.
I'm using a lot of .values to access underlying numpy objects. This will improve some efficiency a bit.
Bug and accidental quadratic runtime
for lat_check in lat_list_str_copy_1:
if letter & set(lat_check):
lat_list_str3 = [i.replace('-', 'S') for i in lat_list_str_copy_1]
lat_list_str3 = ['N' + lat_addchar for lat_addchar in lat_list_str_copy_1]
This code (which occurs with variations four times in your program) is probably ...
There are a few quick improvements you can make. First, always remove as many things as possible from for loops. In this case, the date formatting and the open file lines can be removed.
Dates. Format the dates in your dataframe before the first for loop with something like
df['Date'] = pd.to_datetime(df['Date']).dt.date
Notice I’m converting the datetime ...
Some general tips
Put code in functions
That way you can test each part individually. Here the generation of the sequence, calculation of the ratio and exporting to pandas are clear divisions in the functionality of your code
The best algorithms implement fibonacci sequences as generators
a, b = 0, 1
for _ in ...
You are correct in thinking that iterrows is a very bad sign for Pandas code. Even worse is building up a DataFrame one row at a time like this with pd.concat - the performance implications are dreadful.
Instead of reaching for loops, your first step should be to check if there is a vectorized DataFrame method you could use. In this case.. perhaps there isn'...
You should decide if your functions modify the object they receive or if the return a modified object. Doing both is just asking for disaster. After your code has finished, file_1, file_2 and file_3 are all identical.
The usual convention is to return None (implicitly or explicitly) if you mutate any of the inputs. In the rest of this answer I have decided ...
First, starting from your code, you should realize that you are repeating yourself, three times. This goes against the principle Don't repeat Yourself (DRY).
The only real difference between processing your three sheets are their name and the underlying dataframe, so you could make this into two functions:
from win32com.client import Dispatch
import pandas ...