Extending Opnet Modeler with Client Profiles for Selecting Data Sources in WAN

Introduction

This is a part of interim report of the  Nebula project that focuses on the utilization of WAN infrastructure. One of the purposes of this project is customize the basic functionality of OPNET Modeler 9.0  in order to manage client profiles and monitor network traffic. In particular, we have experimented with 

• Co-simulation with an external program
• Dynamic server switching based on response time

The work has been conducted by  Katsura Fujita under supervision of Vladimir Zadorozhny.

Co-simulation with an external program

Purpose

The purpose of co-simulation is that we are planning to use the Java Handle Performance Monitor Client program with OPNET Modeler's scenario. We created a simple HTTP protocol model to communicate between an external program and co-simulation. The co-simulation API enables to develop the control-flow and data-exchange functions. We referred OPNET WORK 2002 Session: 1532 Interfacising Multiple Simulators using Modelers's Co-Simulation API. The co-simulation model enables us to integrate OPNET into existing environments. 

Scenario

There are six steps to implement the co-simulation:

• External System Definition Model
• Simulation Description
• Esys Module
• Pakcet Format
• Process Model
• External Program

External System Definition

An external system definition defines eys interfaces that will communicate between co-simulation and an external program. As we create an esys interface, we can use a collection of kernel procedures to read values from and write write values to esys interfaces.

Steps to create an external system definition

Choose File New from the Modeler menu bar and select External System Definition from the pull-down list. Click in the Simulator Description.(I'll explain about the simulator description next.) Define any esys interfaces needed by the external simulator, as described in the following section. Add to the Comments field any information a user may need about how to use the ESD model. Save the model.

Steps to create an external interface

Open an ESD model for which you will define interfaces. Click in the Name cell of the first empty row and enter a name for the interface. Click on the Type cell for the interface and select a data type from the list. Click on the Direction cell for the interface and select the data direction. The direction specifies how data flows through the interface. OPNET to Cosim (the esys interface sends data from OPNET to the external code) Cosim to OPNET (the esys interface receives data from the external code) Bidirectional (the esys interface sends and receives data) Not connected (the esys interface is defined, but not used) For a standard esys interface, leave the Dimension attribute set to 0. Click in the Comment cell for the interface and enter a description of the interface.

we created the definition in the table below.

Name	Type	Direction	Dimention
if_from	pointer	Cosim to OPNET	0
if_to	pointer	OPNET to Cosim	0

Simulation Description

When you invoke a cosimulation, for the simulation program, OPNET uses information from a simulation description to find an external program. In addition, OPNET finds attributes such as compilation option, object file names and the type of operating system to invoke a cosimulation from a simulation description. A simulation description consists of one or more blocks of statements set off by the identifiers start_definition and end_definition.

Simulation Description File Statements

Statement	Description	Value
platform	Intended platform for the cosimulation	solaris, windows
kernel	Intended kernel type	development, optimized
dll_lib	Cosimulation DLL library	file name of a library for the cosimulation
bind_lib	Bind arguments	arguments as used on a command line
bind_objs	Bind objects	object file to be bound
use_esa_main	Whether to invoke an esa_main() routine	yes, no

The simulation description file should be put under [mod_dirs] directory with ".sd" suffix.
Note that [mod_dirs] directory can be changed with Preference on main menu.

Simulation Description: cosim.sd

start_definition platform: windows bind_objs: cosim.obj use_esa_main: no end_definition

Esys Module

This is a module on the Node Model and for a gateway between OPNET models and an external code.

Steps to add an esys module

Open Node Model. Choose File. It is possible to create your own node model from scratch, but we recommend you copy an exsisting node from OPNET. In my case, we selected "ehternet_wkstn_adv" as a template. Create Esys Module. There is an icon of esys module on the menu bar. Drag it into the Node Model. Change Attribute. Open esys module's attribute and change esd model into what you define on External System Definition.

Packet Format

There are two packets which we need to trace. One is data packet that we contain our own data, and the other is application packet that OPNET application uses such as HTTP, Ftp or E-mail. To define packet format, Packet Editor enables to define internal structure of packets such as unique name, data type, default value, size in bits, an encoding style, a conversition method and optional comments.
In this scenario, we created a user-defined packet. The user-defined packet contains only one field with the following attributes.

name	data
type	structure
encoding	signed, big endian
size	32

Process Model

First of all, we created the packet generator based on pkgen process model with OPNET WORK 1532.

Packet *	pkptr;     /* OPNET predefined structure */
Packet_Data *	sptr;      /* int a, double b, char c[100] */

pkptr = op_pk_create_fmt ("cosim");

sptr = (Packet_Data *)op_prg_mem_alloc (sizeof (Packet_Data));

sptr->a = op_id_self ();
sptr->b = op_sim_time ();
sprintf (sptr->c, "%s", "Initial Data");

op_pk_nfd_set (pkptr, "data", sptr, 
	op_prg_mem_copy_create, op_prg_mem_free, sizeof (Packet_Data));

op_pk_send (pkptr, 0);

line	description
pkptr = op_pk_create_fmt("cosim");	pkptr is a pointer of OPNET Packet structure. This value is the name of a packet format created using the Packet Format Editor. The packet format specified must be located in the [mod_dirs] directory, or an error will occur. Packet format files have the suffix ".pk.m" appended to the format name.
sptr = (Packet_Data *)op_prg_mem_alloc (sizeof (Packet_Data));	In this case, sptr is a packet data, that will be attached with pkpktr. (pkptr has only meta data information to send or receive a packet) Typical allocations of memory for data structures can use the C language pseudo-function sizeof() to yield the size of the structure to be allocated.
sptr->a = op_id_self ();	Assigns the Object ID into the Packet_Structure. Object ID of the surrounding processor or queue. The special value represented by the symbolic constant OPC_OBJID_INVALID will be returned if this Kernel Procedure is invoked from a non-process context.
sptr->b = op_sim_time ();	Assigns the current simulation starting time into the Packet_Data structure. This double-precision floating-point number represents the simulation time in units of seconds, starting with the begin simulation interrupt, at which point op_sim_time() returns a value of zero.
op_pk_nfd_set (pkptr, "data", sptr, op_prg_mem_copy_create, op_prg_mem_free, sizeof (Packet_Data));	Assigns the value of a sturcture field in the specified packet, in addition to structure copying and deallocation procedures; the field-of-interest is specified by name.
op_pk_send (pkptr, 0);	Forwards the specified packet through an output packet stream, schedules the packet arrival at a destination module for the current simulation time, and releases ownership of the packet by the invoking process.

Second, we modified the esys(esys module) to communicate with an external program.

Packet *pkptr;
Packet_Data *sptr;

switch (op_intrpt_type ())
{
  /* Stream interrupt: packet from source, to be sent to cosim */
case OPC_INTRPT_STRM:
  {
    pkptr = op_pk_get (op_intrpt_strm ());

    sptr = (Packet_Data *)op_prg_mem_alloc(sizeof(Packet_Data));

    op_pk_nfd_get (pkptr, "data", &sptr);
    op_pk_destroy (pkptr);

    /* Write data to interface to trigger external callback */
    op_esys_interface_value_set (interface, OPC_ESYS_NOTIFY_IMMEDIATELY, sptr, 0);

    break;
  }
  /* esys interrupt: data from external code, forward on */
case OPC_INTRPT_ESYS_INTERFACE:
  {

    op_esys_interface_value_get (op_intrpt_esys_interface(), (void *)&sptr, 0);

    pkptr = op_pk_create_fmt ("cosim");
    op_pk_nfd_set (pkptr, "data", sptr,
                   op_prg_mem_copy_create, op_prg_mem_free, sizeof (Packet_Data));
    op_prg_mem_free (sptr);

    op_pk_send (pkptr, 0);
    break;
  }
}

line	description
switch (op_intrpt_type ())	This returns numeric code associated with the current interrupt. There are two interrupts that we can detect output from pkgen(OPC_INTRPT_STRM) and an external program(OPC_INTRPT_ESYS_INTERFACE).
op_esys_interface_value_set (interface, OPC_ESYS_NOTIFY_IMMEDIATELY, sptr, 0);	This procedure places an event on the OPNET event list. At the scheduled time, value appears on the esys interface or vector interface element specified. You can get the esys interface ID from op_topo_child().
op_esys_interface_value_get (op_intrpt_esys_interface(), (void *)&sptr, 0);	This procedure obtains a single value from an esys interface and places it into a variable. The value of the esys interface, or element in the case of a vector interface, remains unaffected.

Finally, we created the process models that encapsulate OPNET application packets to send, and decapsulate TCP packets from the server.
There are four process models, that manage to comunicate with the TCP layer: INIT, OPEN, DATA and RELEASE.

INIT: The first process constructs a TCP packet and retrives attributes from the Process Model Edior. The attributues can be set up [Interfaces] -> [Model Attributes] on the main menu. In my case, we prepared the following attributes.

Attribute Name	Type	Default Value
Packet Size PDF	String	expnential
Packet Size Args	String	8192
Start Time	double	5.0
End Time	double	3,600
Local Port	integer	1024
Remote Port	integer	80
Remote IP Address	String	192.168.0.2
Packet Interarrival PDF	String	exponential
Packet Interarrival Args	String	5.0

*Packet *upper_pkptr = op_pk_get(op_intrpt_strm ());

my_id = op_id_self ();
sprintf (pid_string, "tcp_lab_gen PID (%d)", op_pro_id (op_pro_self ()));

/* Load the distribution used to determine the size of the */
/* packets being generated.                                */
dval0 = dval1 = 0.0;
op_ima_obj_attr_get (my_id, "Packet Size PDF", pdf_name);
op_ima_obj_attr_get (my_id, "Packet Size Args", pdf_args);
sscanf (pdf_args, "%lf %lf", &dval0, &dval1);
packet_size_distptr = op_dist_load (pdf_name, dval0, dval1);

/* Load a distribution used to determine the frequencey of */
/* packet generation.                                      */
dval0 = dval1 = 0.0;
op_ima_obj_attr_get (my_id, "Packet Interarrival PDF", pdf_name);
op_ima_obj_attr_get (my_id, "Packet Interarrival Args", pdf_args);
sscanf (pdf_args, "%lf %lf", &dval0, &dval1);
packet_interarrival_distptr = op_dist_load (pdf_name, dval0, dval1);

op_ima_obj_attr_get (my_id, "Start Time", &conn_start_time);
op_ima_obj_attr_get (my_id, "End Time", &conn_end_time);
op_ima_obj_attr_get (my_id, "Local Port", &local_port);
op_ima_obj_attr_get (my_id, "Remote Port", &remote_port);
op_ima_obj_attr_get (my_id, "Remote IP Address", ip_addr_str);
/*op_ima_obj_attr_get (my_id, "Remote IP Address", "192.168.0.2");*/

op_intrpt_schedule_self (conn_start_time, TCP_LAB_GEN_CONN_OPEN);

ip_addr = ip_address_create (ip_addr_str);

tcp_app_handle = tcp_app_register (my_id);
op_pk_send(upper_pkptr, 0);

line	description
op_ima_obj_attr_get (objid, attr_name, value_ptr)	Provides a mechanism to get attributes during the simulation. The first argument, objid, is retrieved by op_id_self(). attr_name and value_ptr are set up in the Model Attributes as we described before.
op_intrpt_schedule_self (time, code)	Schedules the start of operation. The new event represents a self interrupt that is scheduled to be delivered at the time. code is an arbitrary, user-defined integer which can be used as an identification code for the scheduled interrupt.
tcp_app_register (objid)	Registers application layer with the API package. This returns an handle which contains connection inforamtion on all subsequent calls to the API.

OPEN: The interrupt indicates that an OPEN command should be issued to the TCP Layer to open a connection. Also, schedules a self-interrupt for sending the CLOSE command.

if (intrpt_code == TCP_LAB_GEN_CONN_OPEN)
	{
	connect_id = tcp_connection_open 
		(&tcp_app_handle, ip_address_copy (ip_addr), remote_port, local_port, TCPC_COMMAND_OPEN_ACTIVE, 0);

	op_intrpt_schedule_self (conn_end_time, TCP_LAB_GEN_CONN_CLOSE);
	}

line	description
tcp_connection_open (&tcp_app_handle, ip_address_copy (ip_addr), remote_port, local_port, TCPC_COMMAND_OPEN_ACTIVE, 0);	Open a TCP connection between the host and a remote host.

DATA: This process model manages to encapusulate the upper layer's packets and to destroy incoming TCP packets.

Packet *upper_pkptr;

if (CONN_ESTABLISHED || PKT_SEND)
{
  /* Create a packet using the outcome of the loaded */
  /* distribution.                                   */
  pksize = op_dist_outcome (packet_size_distptr);
  pkptr  = op_pk_create (pksize);

  /* create upper layer packet */
  upper_pkptr = op_pk_get(op_intrpt_strm ());

  /* encapsulate the packet into original packet */
  op_pk_fd_set(pkptr, 0, OPC_FIELD_TYPE_PACKET, upper_pkptr, -1);

  tcp_data_send (tcp_app_handle, pkptr);

  tcp_receive_command_send (tcp_app_handle, 1);

  /* Schedule a self-interrupt so that the client */
  /* will send another packet.                    */
  next_packet_arrival_time = op_sim_time () + op_dist_outcome (packet_interarrival_distptr);
  op_intrpt_schedule_self (next_packet_arrival_time, TCP_LAB_GEN_PK_CREATE);
}

if (PKT_RECEIVE)
{
  /* Destroy the received packet(s) and ICI. */
  op_ici_destroy (op_intrpt_ici ());
  op_strm_flush (OPC_STRM_ALL);
}

RELEASE: The interrupt indicates that the connection need to be closed. Issue a CLOSE ccommand to close this end of the connection.

if (CONN_RELEASE)
	{
	pkptr = op_pk_create (8);
	op_pk_fd_set (pkptr, 1, OPC_FIELD_TYPE_INTEGER, 1, 0);

	tcp_data_send (tcp_app_handle, pkptr);
	tcp_connection_close (tcp_app_handle);
	}

External Program

The code is for a external program. The source code should be put under [mod_dirs] directory. The program 1)retrieves data from OPNET, 2)modify the data, 3)and send it back to the OPNET. The program is executable linked using op_mksim command.
The following steps are to build and run the co-simulation.

1. cl /c cosim.c /I. /IC:\OPNET\9.0.A\sys\include (Build the external object file)
2. op_mksim -net_name project_name-scenario_name (Build the static simulation)

cosim.c

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include "esa.h"
#include "cosim.h"

EsaT_State_Handle esa_handle;
EsaT_Interface interface_out;

void new_value_callback(void *state_ptr, double time, va_list vararg);

int main(int argc, char *argv[])
{
  double           time_val;
  int              exec_options;
  int              num_events;
  double           time_reached;
  int              num_interfaces;
  EsaT_Interface   *interfaces;
  int              status, i;
  const char       *if_name;
  char             *dot;

  Esa_Main(argc, argv, 0);  /* 0 = don't look for and invoke esa_main() */
  Esa_Init(argc, argv, ESAC_OPTS_NONE, &esa_handle);
  Esa_Load(esa_handle, ESAC_OPTS_NONE); /* load and set up an OPNET simulation repository */

  Esa_Interface_Group_Get(esa_handle, &interfaces, &num_interfaces);

  for (i = 0; i < num_interfaces; i++) {

	if_name = Esa_Interface_Name_Get(esa_handle, interfaces[i]);
	dot = strrchr(if_name, '.');
	if (dot != NULL) {
	  if (!strcmp(dot, ".if_to")) {
		Esa_Interface_Callback_Register(esa_handle, &status, interfaces[i], new_value_callback,	(EsaT_Interface_Array_Callback_Proc)NULL, (void *)NULL);
	  } else if (!strcmp(dot, ".if_from")) {
		interface_out = interfaces[i];
	  }
	}
  }

  /* Advance OPNET sim based on requested options */
  time_val = 100;
  exec_options = ESAC_UNTIL_INCLUSIVE;
  Esa_Execute_Until(esa_handle, &status,
					time_val, exec_options, &time_reached,
					&num_events);

  Esa_Terminate(esa_handle, ESAC_TERMINATE_NORMAL);

  if (status == ESAC_STATUS_TERMINATION) {
	exit(-1);
  }
}


void new_value_callback(void *state_ptr, double time, va_list vararg)
{
  /* A new data structure has been provided by the OPNET sim      */
  /* Just schedule its delivery back to OPNET at time+n seconds   */
  Exchanged_Data *info;
  EsaT_Compcode retval;
  int status;

  info = va_arg(vararg, Exchanged_Data *);
  sprintf(info->data->c, "%s", "Additional Data");

  /* *ADD* callback processing code */
  retval = Esa_Interface_Value_Set(esa_handle, &status, interface_out, time, info);
}

Debugging the simulation

Preparing for Debugging

• Set flags to include debuggin inforamtion under [Edit] -> [Preference]
• Add "/Od /Zi" to comp_flags
• Add "/DEBUG" to bind_shobj_flags
• Allow debugger to handle exception instead of OPNET
• Force model recompilation.(See the figure.)
• Recompile process models

Debugging OPNET simulation

• Run simulation
• Press Cancel to run debugger when there is an error message
• Find the value in the invalid address

Tracing OPNET simulation

• Open simulation dialog
• Select 'Use OPNET Debugger'
• Run Simulation (simulation will stop after compilation)
• Run Visual Studio
• Select [Tool] -> [Debug Processes]
• Select "op_runsim.exe"
• Push Attach Button
• Select [Debug] -> [New Breakpoint]
• Input the value or function name to break
• Type 'run' in the simulation console to run the simulation

Result and Problem

Generate data packets from the external program

Recive packets from TCP module

Dynamic switching servers based on response time

Purpose

The purpose of the dynamic switching servers is that we put the swing algorithm in the process model, wihch changes the destination based on the response time from servers. Once the response time from one server delays, it can switch the other one.

Scenario

We created a simple scenario, which has one client and two servers. Both severes provide HTTP service defined by Application Config.
We modified the gna_http_mgr_spawn_session () in gna_http_mgr process model and sent_traffic_stats_write () in gna_http_cli process model.
gna_http_mgr process model figures out when and where to send the HTTP request to, then spawns a gna_http_cli process to do the actual work. Therefore to change the destination of the traffic, gna_http_mgr process model is needed to modify the code in gna_http_mgr process model.
In this process there is a function named 'gna_http_mgr_spawn_session ()' in the function block, that creates the HTTP client. One of the input is 'server_index', which is the index of destinations. We created the list of response time from send_traffic_stats_write () in gna_http_cli process model and wrote it into the file. This enables gna_http_mgr process model read response time from each servers.
There are a few places where this is used, some for HTTP v1.0 and some for HTTP v1.1. All of these use the function 'gna_http_mgr_server_pick ()' tp determine the destination of the traffic, so we changed the 'server_index' after this function.

Process Model

received_traffic_stats_write() in the gna_http_cli process model

Write server name, response time and order into the log.

	/* Select server for the inline object */
	serv_index = gna_http_mgr_server_pick (http_page_ptr->object_servers_ptr [obj_index]->appl_servers_ptr, 
	http_page_ptr->object_servers_ptr [obj_index]->server_indexes_lptr, 
	http_page_ptr->object_servers_ptr [obj_index]->number_of_servers, INLINE_PAGE);

....
	/* Get response time */
	obj_resp_time = op_sim_time () - op_pk_stamp_time_get (pkptr);
	http_page_ptr = (GnaT_Cli_Http_Page *)op_pro_parmem_access();

	/* Get server name */
	op_pk_nfd_access (pkptr, "dest server name", &server_name);

	/* check the order from the log file */
	if ((fp = fopen("C:\\Home\\log.txt", "r+")) == NULL)
		{
		count = 0;
		}
	else
		{
		while ((ch = fgetc(fp)) != EOF) {
			if (ch == '\n')
				count++;
			}
		fclose(fp);
		}
	
	/* write server name, response time and order into the log */
	if ((fp = fopen("C:\\Home\\log.txt", "a+")) == NULL)
		{
		printf("(katsura cli) can't open log.txt.\n");
		}
	else
		{
		fprintf(fp, "%s %f %d\n", server_name, obj_resp_time, count);
		fclose(fp);
		}

gna_http_mgr_http11_inline_obj_request() in the gna_http_mgr process model

Note that serv_index2 is a global value and it'll change '1' or '0' everytime packet comes.

	/* Make sure that we have at least one inlined object to send a request for */
	if (total_number_of_objects > 0)
	{					
	/* Loop through all servers and start fetching inline objects */
	for (serv_index = 0; serv_index <= max_server_index ; serv_index++)
	{
		if (server_info_array_ptr [serv_index]->send_objects_lptr != OPC_NIL)
		{
			if (op_prg_list_size (server_info_array_ptr [serv_index]->send_objects_lptr) > 0)
			{
				/* Check whether a TCP connection is already opened to this server */
				session_ptr = gna_http_mgr_connection_check (serv_index, INLINE_PAGE);
		
				/* Set flags for the client session */
				cli_command = HTTP_RTRV;
				mgr_command = REUSE_CONNECTION;
				/* Check if connection has not yet expired */
				if (session_ptr != OPC_NIL)
				{
					/* Session has expired */
					if (session_ptr->expire_time < op_sim_time ())
					{
						op_prg_list_free(session_ptr->objects_queued_lptr);
						op_prg_mem_free(session_ptr->service);
						op_prg_mem_free(session_ptr);
						session_ptr = OPC_NIL;
						mgr_command = NEW_CONNECTION;
					}
				}				
				/* Spawn a TCP session for each server */

				/* modification starts */				
				
				/* switch the server */
				if (serv_index == max_server_index)
				{
					serv_index2 = serv_index - 1;
				}
				else
				{
					serv_index2 = serv_index + 1;
				}
				/* modification ends */

				gna_http_mgr_spawn_session (session_ptr, mgr_command, serv_index2, cli_command,
								server_info_array_ptr [serv_index]->send_objects_lptr);
			}
		}
	}

Result and Problem

Log result

server2 0.001171 0
server1 0.002167 1
server1 0.129152 2
server1 0.002167 3
server1 0.129151 4
server2 0.001171 5
server1 0.001170 6
server1 0.001170 7
server2 0.002167 8
server2 0.129152 9
server1 0.001200 10
server1 0.002167 11
server1 0.129152 12
server1 0.001170 13
server1 0.001160 14
server1 0.001170 15
server2 0.001171 16
server2 0.001171 17
server1 0.002167 18
server1 0.129152 19
server2 0.001201 20
server2 0.001170 21
server2 0.001161 22
server1 0.001170 23
server2 0.002167 24
server2 0.129152 25
server2 0.001171 26
server1 0.002166 27
server1 0.129152 28
server2 0.001201 29
server1 0.001170 30
server2 0.001171 31
server1 0.001170 32
server1 0.001170 33
server2 0.002167 34
server2 0.129152 35
server2 0.001171 36
server2 0.001160 37
server2 0.001171 38
server1 0.001170 39
server2 0.001170 40
server2 0.002167 41
server2 0.129152 42
server1 0.001170 43
server2 0.001171 44
server1 0.001170 45
server2 0.001171 46
server1 0.001170 47
server1 0.001170 48
server1 0.001160 49
server2 0.002167 50
server2 0.129152 51
server2 0.002167 52
server2 0.129152 53
server1 0.001170 54
server2 0.001170 55
server1 0.001170 56
.....
server2 0.129152 147
server1 0.001200 148
server2 0.001171 149
server1 0.001170 150
server2 0.001171 151
server1 0.001170 152
server1 0.002167 153
server1 0.129152 154
server1 0.001170 155
server1 0.001160 156
server1 0.001170 157
server2 0.001170 158
server1 0.001171 159
server2 0.001170 160
server2 0.001170 161
server1 0.002167 162
server1 0.129152 163
server2 0.001201 164
server1 0.001170 165
server1 0.002167 166
server1 0.129152 167

The client could switch the destinations based on the list of the response time. Howerver, there is one problem that the result was not what we expected before. If we switch the servers in turn, we expect that the order of response from servers is also in turn. The OPNET creates some inline objects even if we specify only one HTTP object in Application Config. The sequence of response time from a same server is correspond to the number of inline objects.
We figured out that there is a lot of things going on in all of the protocols so the order at which things are sent is not necesarilly the order in which the response are seen. For eaxmple if going through a switch the second request may experience a small extra delay in the switch, then the processing time at the servers may be different.

The client could switch the destinations based on the list of the response time. Howerver, there is one problem that the result was not what we expected before. If we switch the servers in turn, we expect that the order of response from servers is also in turn. The OPNET creates some inline objects even if we specify only one HTTP object in Application Config. The sequence of response time from a same server is correspond to the number of inline objects.
We figured out that there is a lot of things going on in all of the protocols so the order at which things are sent is not necesarilly the order in which the response are seen. For eaxmple if going through a switch the second request may experience a small extra delay in the switch, then the processing time at the servers may be different.