There are a few performance tricks we can apply here:
add __slots__ to the class definition should help with memory and performance as well:
"""Set up regular expressions"""
# use https://regexper.com to visualise these if required
_reg_school = re.compile(r'School = (.*)\n')
_reg_grade = re.compile(r'Grade = (.*)\n')
I would try to use existing pandas features where possible to keep this code minimal - this aids readability and reduces the possibility of bugs being introduced in complicated loop structures.
from scipy.stats import chi2_contingency
def chisq_of_df_cols(df, c1, c2):
groupsizes = df.groupby([c1, c2]).size()
ctsum = groupsizes.unstack(...
Let's start with the obvious: this code doesn't run. You're missing ans = starter() so that further (el)if ans.lower() == ... doesn't miserably fail with a NameError.
Likely, you define q() but never use it.
And you also appears to have other useless stuff floating around: why use both textblob and indicoio to perform sentiment analysis? You also seem to ...
I think you need create boolean DataFrame by compare all filtered columns values by scalar for not equality and then check all Trues per rows by all:
df = df[(df[['A','C']] != 0).all(axis=1)]
A B C
0 1 2 5
2 6 8 4
print (df[['A','C']] != 0)
0 True True
1 True False
2 True True
3 False True
You’re using the wrong tool for the job. Basically, you do all the computation in Python, use numpy for intermediate storage and pandas for display.
Instead, you should compute the list of tribonacci numbers and from there on use pandas for anything else as it would be much more efficient / readable. I’d keep building the tribonacci numbers in Python as I ...
I've never used numpy or matplotlib, so I can only speak to issues of style.
You're allowing for far too much nesting here. Your code consists of a giant, dense, deeply nested chunk. As a result, the eyes have very few good landmarks to rest on; making your code hard to read.
Put your existing function definitions at the very top level outside of any ...
AFAIK it does what you want (but on this site you should generally be
sure that your code does what you want beforehand).
Beauty, eye of the
...; that said, this code can be rewritten in a very concise manner:
import pandas as pd
import scipy.stats as scs
return range(int(series.min()), int(series.max()) + 1)
I'm going to put this code into an editor, "proofread" it from top to bottom, give notes as I go, and then paste the final result to show the effect of the edits.
Code editors don't usually wrap lines since linebreaks mean things in code. Consequently, most coding style guidelines suggest limiting your column width so the reader doesn't need to scroll ...
You have eight conditions to match for every UPDATE. A typical solution would store timestamps using a DATETIME or TIMESTAMP column, so that there is only one value to match.
For reasonable performance, ensure that the timestamp field is indexed.
Using crosstab, this can be done in a single step:
It will give this desired result:
var1 | 0 1
0 |0 1
1 |2 0
2 |2 0
You can name the index and colnames and also get the row totals and column totals:
It doesn't look like you really need regular expressions. This construct just using basic string operations is about 10x faster than the construct with the regular expressions:
present_colors = set()
for value in ex_df['colors'].values:
for color in [x.strip() for x in value.split(';')]:
And a bit ...
You take the combinations in a way too convoluted way. For starter, I would simplify the retrieval of "same memberid" questions:
for memberid in IncorrectQuestions.memberid.unique():
questions = IncorrectQuestions[IncorrectQuestions.memberid == memberid].questionid
No need to groupby here since you’re already filtering by the one memberid you’re ...
Iterating in Python is slow, iterating in C is fast. So, it's best to keep as much as possible within Pandas to take advantage of its C implementation and avoid Python.
You can use apply on groupby objects to apply a function over every group in Pandas instead of iterating over them individually in Python.
df["metric1_ewm"] = df.groupby("person").apply(...
I am the Adams of Adams-Skopek. I want to point out a couple of things:
You get some truncation error when you sum or multiply a series of values in different orders. You need to estimate this and account for it.
There is a lower limit on sparsity of the data, you need to address this.
If you can get a copy of the Pagano and Halverson program (cited in ...
Setting aside PEP 8 (official style guide) issues I would make the following change:
Rather than keeping your politician bins, names and email addresses in separate data structures (the only way they are tied right now is by the array index number) - I would create a Politician class. Something like:
def __init__(self, name, ...
Imagine how much of a pain it would be to add another politician to this list. In addition to the Politician class suggested in another answer, I would suggest creating a list of Politicians and whenever you want to do something to one of them, iterate over this list. This may require adding a method to your Politician class. Then, adding another candidate ...
nlargest could help, it finds the maximum n values in pandas series.
In this line of code, groupby groups the frame according to state name, then apply finds the 3 largest values in column CENSUS2010POP and sums them up. The resulting series has the unique state names as index and the corresponding top 3 counties sum, applying nlargest gets a series with ...
Bug and accidental quadratic runtime
for lat_check in lat_list_str_copy_1:
if letter & set(lat_check):
lat_list_str3 = [i.replace('-', 'S') for i in lat_list_str_copy_1]
lat_list_str3 = ['N' + lat_addchar for lat_addchar in lat_list_str_copy_1]
This code (which occurs with variations four times in your program) is probably ...
As you already use pandas, you are right that there is no overhead on using Timestamps (the kind of objects returned by pd.to_datetime). But, as said in the documentation (help(pd.Timestamp)):
Help on class Timestamp in module pandas.tslib:
| TimeStamp is the pandas equivalent of python's Datetime
| and is ...
There are a few quick improvements you can make. First, always remove as many things as possible from for loops. In this case, the date formatting and the open file lines can be removed.
Dates. Format the dates in your dataframe before the first for loop with something like
df['Date'] = pd.to_datetime(df['Date']).dt.date
Notice I’m converting the datetime ...
Some general tips
Put code in functions
That way you can test each part individually. Here the generation of the sequence, calculation of the ratio and exporting to pandas are clear divisions in the functionality of your code
The best algorithms implement fibonacci sequences as generators
a, b = 0, 1
for _ in ...
You are correct in thinking that iterrows is a very bad sign for Pandas code. Even worse is building up a DataFrame one row at a time like this with pd.concat - the performance implications are dreadful.
Instead of reaching for loops, your first step should be to check if there is a vectorized DataFrame method you could use. In this case.. perhaps there isn'...
You should decide if your functions modify the object they receive or if the return a modified object. Doing both is just asking for disaster. After your code has finished, file_1, file_2 and file_3 are all identical.
The usual convention is to return None (implicitly or explicitly) if you mutate any of the inputs. In the rest of this answer I have decided ...
The usual approach -- if you want all the projects -- would be
Name: HOURS, dtype: float64
This only applies the sum on the desired column, as this constructs an intermediate SeriesGroupBy object:
Without knowing what your csv file is structured, it is hard to give too much concrete. I do have some suggestions, however.
You can most likely drastically increase performance by converting strings like the player names to categorical data. Strings are slow in pandas, especially string lookup in a large column (as you have here many times). Using ...
In the while loop, the first call you make to nmf_model.fit_transform() is superfluous and can be removed. You aren't even using the results of the transformation calculation. The next line, where you have W = nmf_model.fit_transform(X_imputed.values) is doing all the work. Removing this line halves the number of model fits and speeds things up by ~...
DISCLAIMER: I don't know cython and have never used it, so if any of my advice doesn't apply because of cython limitations, feel free to disregard it.
Counting neighbors in a game board is very easy to do via convolution with the proper kernel. Below, I used SciPy's convolve function, not the much faster fftconvolve, because the latter only works on floats ...
Use a class, or at least some functions, to make your code more readable and understandable
Very first reaction looking at your code is ....blech. I don't want to read that giant blob.
Why not make a class to bundle a bunch of functions together, such as a function readAndConcatAHT? Actually, many of these for loops are doing the exact same thing for ...
It is good to use a library rather than re-inventing everything yourself. Just be sure to avoid explicit looping in Python:
sum(date.day == 1 and date.weekday() == 6 for date in rng)
The above sums the number of times that date.day == 1 and date.weekday() == 6 automatically, with no loops of counters, It should also be more efficient (sum is implemented in ...