I define a class that we will call here State, describing the physical state of my system.The State class has a method at(time) such that State.at(time) models the dynamic behaviour of such system and returns system state at time time.
Time domain is linearly discretised and some calculations are performed at each time step.The characterisation I need is completely based on the system states at the beginning and at the end of the time step, and is performed by the function get_data(state_start, state_end)
The code would run as follows:
from options import initial_time, final_time, number_of_stepsfrom processlib import get_dataimport numpy as np# Define discretised time:time_domain = np.linspace(initial_time, final_time, number_of_steps+ 1, endpoint=True)# Initialise output:out = []# Initialise iteration:time_start = time_domain[0]state_start = State.at(time_start)# Loop over each time step:for time in time_domain[1:]: # Define time step final time: time_end = time # Define start and end states: state_end = State.at(time_end) # *** # Perform calculations: processed_data = get_data(state_start, state_end) # Store results: out.append(processed_data) # Pre-process next iteration by using same data: state_start = state_endprint(out)Using the states at each time step extrema, allows to save computational time by computing the end state of a time step and the initial state of the following one, only once.
The State dataset i am using is very large so this behaviour is very desirble, but it makes it hard to parellelise the process. Also, I cannot fully propagate the system prior to processing the data using State.at(time_domain), due to memory issues.
Nevertheless, in principle, as soon as the line state_end = State.at(time_end) (highlighted by the markers ***) is exectued, the next iteration could already start on a different thread. Is there a way to implement this?
My current knowledge about parallelisation is limited to creating a parallel pool and use multi-thread on loops with independent iterations.The only solution I could come up with is to use basic parallelisation and having to compute the same state twice.