Prompt Engineering in ChatGPT

At the beginning of my career, once I had an issue. I asked another expert in the team for fixing the issue. He also did not know how to resolve it, and he did Google Search. He got the results and fixed the issue.

After he left, I thought, I also did google search, but could not get the correct results. Maybe, I need to learn how to do google search correctly. I have also read a few articles about searching in a better way in google, but did not make much difference. (Back then courses for each and every topic were not that much popular.)

It took me quite some time to understand that, there was nothing to learn in  "Google Search", but, I need to learn the domain.

Whenever I see Prompt Engineering courses for ChatGPT, I exactly feel the same thing.

Eventhough both the expert and myself did not know the answer to the issue, but, we both were not at the same level. He knew everything around the issue except the issue, and I did not know anything. 

When somebody knows everything around the issue, many times they just need the approach to go in the right direction to solve it. Sometimes, even a hint is enough for them. But, for those who do not know anything, those steps are not enough. 

The way an expert ask the question is a lot different from a novice user. Novice user's question would be very generic, and the expert's question would be very specific. Even if both ask the exact same question, the way both would process the same results would be a lot different. As soon as the expert gets a hint or approach, he/she would start working on that. But, a hint/approach is not enough for a novice user.

Whenever I asked any question in the domain which I have decent knowledge, I could use ChatGPT's reply easily and could solve the problem soon.

For any questions for which I did not have any domain knowledge, it was pretty horrible experience. For a few problems, it has taken me a few months to get the optimum solution. [In those months, I was learning the domain.]

In general, I do not have much good opinion on Prompt Engineering courses. But, if I were to give an opinion on that, I would say, treat ChatGPT like a very senior engineer in your organization. When I was a fresher, my then organization gave a session to all the freshers on how to get help and how to ask questions etc. 

If you are going to send a mail to a senior employee (probably in another timezone) asking for help, and sending just a message like "That code is not working" is not correct. 

One should give full context on the problem. What is the issue? In which cases, it occurs, and in which cases, it does not occur? What are the steps you tried? What was the change in the behavior when you tried those steps?

If one does not give full details, it would be difficult for anyone to respond. If anyone asks me for any help without providing any details, I generally ask them all the questions to get the context, and then I help.

Unfortunately, as of today, ChatGPT does not ask the questions back to get the clarity. It would either assumes a few things (and may lead to incorrect solution), or may give solution for all the assumptions (which would be overwhelming).

Also, if we ask ChatGPT multiple independent questions in a single prompt, it does not give efficient answer when compared to asking them separately. Of course, if the questions are related, they should be asked together to give better context. 

If you have absolutely zero knowledge on some domain, and if you are not able to get the correct solution from ChatGPT, I don't think, you can get the solution by doing those prompt courses. But, you can get the solution, when you start understanding the domain (which you can learn from ChatGPT itself). 

How to get the postman collection data of others?

Create a postman collection that is of some interest to other users. 

Invite them to your postman collection. 

Once they accept your invitation, remove them from your team, and while removing, select the option to copy their private workspace to your account.

Voila..  You got full access to their private workspaces, variables and many others. 

At this point, they would lose access to all their workspaces. If you want to appear to be nice, add them back immediately after copying their collections, and grant them access to their workspaces. If they do not pay much attention, they may think that there was some glitch in between and it got resolved automatically. But, in between, you would have got full access to all their collections.

Like the spam mails that are sent, if an invitation mail is sent to all the software engineers, definitely a few of them would click accept and you would get access to all their postman collections.

Typically many people who use postman extensively would keep the passwords, api keys and many other confidential information in their private workspace, assuming nobody could access. If you could tempt them to join your team, you can get all the confidential information. 

However, this would work, only if that person has not yet joined any team. If that person has joined any team, the owner of the first team to which he/she joins, would get the access to that workspace. 

Error in ElasticSearch Cloud - The client is unable to verify that the server is Elasticsearch due to an unsuccessful product check call

When we get the below error, the issue might be with respect to establishing SSL connection with Elastic Search Cloud. 

If the client is connecting to ElasticSearch cloud with old TLS protocol (like 1.1), then you may get this error.

To connect with TLS 1.2, add the following line at appropriate place (if you are using .Net Framework).

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12

If it is a Web application/API, you can add it to Application_Start of Global.asax.

If it is a Console application, you can add it at the beginning or before making the call to ElasticSearch Cloud. 

 Invalid NEST response built from a unsuccessful () low level call on POST: /_bulk

# Invalid Bulk items:

# Audit trail of this API call:

 - [1] ProductCheckOnStartup: Took: 00:00:00.5350041

 - [2] ProductCheckFailure: Node: https://someid.region.azure.elastic-cloud.com:9243/ Took: 00:00:00.5350041

# OriginalException: Elasticsearch.Net.ElasticsearchClientException: The client is unable to verify that the server is Elasticsearch due to an unsuccessful product check call. Some functionality may not be compatible if the server is running an unsupported product. Call: Status code unknown from: GET / ---> Elasticsearch.Net.PipelineException: The client is unable to verify that the server is Elasticsearch due to an unsuccessful product check call. Some functionality may not be compatible if the server is running an unsupported product. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.

   at System.Net.HttpWebRequest.GetResponse()

   at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)

   --- End of inner exception stack trace ---

   at Elasticsearch.Net.RequestPipeline.ThrowIfTransientProductCheckFailure()

   at Elasticsearch.Net.RequestPipeline.Ping(Node node)

   at Elasticsearch.Net.Transport`1.Ping(IRequestPipeline pipeline, Node node)

   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)

   --- End of inner exception stack trace ---

# Request:

<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

# Response:

Elasticsearch - Backup and Restore using Snapshots

The data in Elasticsearch can be backed up using snapshots and it can be restored to a different Elasticsearch cluster.

To take backup of the Elasticsearch on the source cluster, first we need to create a repository (in Elasticsearch) with the target location and type.

The type of the repository is, where do you want to store the backup files. The type can be File Share, Microsoft Azure, Amazon S3, Google Cloud or Hadoop HDFS.

Complete details on Creating a Snapshot is found at https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html

Creating a File Share Repository:

put http://10.21.32.43:9200/_snapshot/mybackup

{
  "type": "fs",
  "settings": {
    "location": "/data/elasticsearch/backup"
  }
}

The above request would create a repository of type File Share and the backup files would be copied to /data/elasticsearch/backup

Creating a Repository in Azure:

Complete details of Azure Repository can be found at https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-azure.html

Before you create a repository in Azure (or even for other providers), please check your access to Azure Repository from your local and from all the Elastic Search Servers. Admin can configure access to cloud storage based on IPs. When there is an IP restriction in Azure, there is no way to find out that from the error messages.

We have to install a plugin for Azure for backup/restore. Command for installing azure plugin is

bin/elasticsearch-plugin install repository-azure

In linux, you may have to use sudo (If not running from root account). elasticsearch may be found at /usr/share/elasticsearch/bin

After installing Azure plugin, we need to add the storage account name and the secret key in Elasticsearch to connect to Azure.

echo This/is/a/key/for/Azure/Storage== | elasticsearch-keystore add -stdin azure.client.default.key

echo AzureStorageAccountName | elasticsearch-keystore add -stdin azure.client.default.account

You may have to use sudo in linux.

Once Elasticsearch is configured to use Azure, we can create a repository.

put http://10.21.32.43:9200/_snapshot/mybackup

{
  "type": "azure",
  "settings": {
    "client": "default",
    "container": "test",
    "base_path": "testpath"
  }
}

This would create a repository, and the snapshot files would be created in testpath in the test container of the storage account.

Taking Snapshot:

The below request starts the process of creating a snapshot

put http://10.21.32.43:9200/_snapshot/mybackup/snapshot1?wait_for_completion=false

With the below request, you can see the status of the snapshot. For any snapshots in progress, it would show the details of the snapshot like how many indices are completed, in progress, not yet started etc. If there are no pending snapshots, it would return an empty array.

put http://10.21.32.43:9200/_snapshot/_status

Incremental Snapshots:

When you take multiple snapshots in a repository, each snapshot is complete for all practical purposes. Any snapshot can be restored or deleted independent of any other snapshot.

But, internally, the snapshots are incremental. i.e., when you take another snapshot in the same repository, it would create the files for the changes that happened since the last snapshot. The metadata of the snapshot would be created appropriately to get the data required from that snapshot.

Creating Read Only Repository (On Destination):

On the destination Elasticsearch cluster, we need to create a read only repository pointing to the same location (either file share or Azure or any other cloud provider). Making the repository as read only is the only difference with respect to the creation of the repository on the source cluster.

put http://10.89.78.67:9200/_snapshot/mybackup

{
  "type": "azure",
  "settings": {
    "client": "default",
    "container": "test",
    "base_path": "testpath",
    "readonly": true
  }
}

Restoring the Snapshots:

The below request would restore the snapshot on the destination cluster.

post http://10.89.78.67:9200/_snapshot/mybackup/snapshot1/_restore

Restoring Status:

There is no specific request to check the status of the restore. However, when the restore is started, the cluster would go to yellow state. Once it goes to Green state, we can understand that the restore is completed.

The below request gives the status of the cluster.

get http://10.89.78.67:9200/_cluster/_health

Restoring Incremental Snapshots:

When you restore incremental snapshots, you would get an error since there is already an index in the destination cluster with the same name as the index that the restore is trying to create.

To restore the snapshot from second time onwards, you need to close the existing indices, so that the cluster can update them.

Request to close all the indices

post http://10.89.78.67:9200/_all/_close

Request to close one index

post http://10.89.78.67:9200/indexName/_close

Restore after Closing the Indices:

During the restore, Elasticsearch would open the indices that needed to be updated. If any index is not there in the snapshot, then that index would not be opened and it would continue to be in closed state.

If you are using the destination cluster only as a backup of another cluster, if you see any index in closed state after a restore, it means that index was deleted from the source cluster.

Handling Aliases:

If you are having aliases in the source cluster, and if the underlying index is getting changed and the old index is getting deleted, then you need to do special handling on the destination cluster.

For example, in the source cluster, you create one index everyday. The index name may be something like IN20201105, IN20201106 etc. You have an alias named INDate, which is pointing to that day's index. You keep only the active index and remove all the old indices.

In this case, when you take snapshots everyday, the indices are going to be different each day. The index that was present yesterday won't be present today.

When you restore the snapshots everyday on the destination cluster, the deleted indices would be in closed state.

When the indices are restored, the aliases would be restored with the relationship with the new index, but it won't delete the relationship with the old index. The alias would be pointing to both old and new indices. When you query the alias, it would have both the old index (that was deleted in the source cluster and closed in the destination cluster) as well as the new index. Since, one index in the alias is in closed state, you would get an error when you are trying to query it.

The simple option is deleting all the indices that are in closed state after a restore.

Request for deleting an index

delete http://10.89.78.67:9200/indexName/

Automated Snapshot and Restore everyday:

The snapshot can be taken in the source cluster without having any impact on the source cluster. However when you restore, the cluster would be down for some time.

You can use any scripting language that you are comfortable for the automation. I used WGET and VIM for automatic snapshot and restore.

There are 4 steps in the entire process of snapshot and restore. First step is on the source cluster and the next 3 steps are on the destination cluster. You have to schedule in such a way that, before the second step is started, the first step is completed.

1. Take Backup in the source cluster

Request in Linux

wget -d --method=put "http://10.21.32.43:9200/_snapshot/prodbackup/s`date +\%Y\%m\%d`?wait_for_completion=false" -O CreateSnapshotES.txt -o CreateSnapshotNetwork.txt

Request in Windows

In different versions of Windows (and/or localization), the date command is displayed with different formats. You need to check the date command's format.

wget -d --method=put "http://10.21.32.43:9200/_snapshot/prodbackup/s%DATE:~-4%%date:~4,2%%date:~7,2%?wait_for_completion=false" -O CreateSnapshotES.txt -o CreateSnapshotNetwork.txt

2. Close the indices on the target cluster

Request to close all the indices

wget -d --method=post "http://10.21.32.43:9200/_all/_close" -O CloseDRIndicesES.txt -o CloseDRIndicesNetwork.txt

3. Restore the snapshot on the target cluster

Request to restore the snapshot on the target cluster [Date part of the request should be same as in the request 1.]

wget -d --method=post "http://10.89.78.67:9200/_snapshot/prodbackup/s`date +\%Y\%m\%d`/_restore" -O DRRestoreES.txt -o DRRestoreNetwork.txt

4. Delete the closed indices

I used VIM and WGET for automating this.

Create a file vimclose.vim with the following content (Change the IP address in your script). Make sure that there is a new line at the end (after :wq).

:v/close/d
:%s/.*close *//
:%s/  *.*//
:%s/\(.*\)/wget -d --method=delete "http:\/\/10.89.78.67:9200\/\1" -O \1.json -o \1.txt/
:wq

Create a batch file or shell script with the following content.

For Windows.

wget "http://10.89.78.67:9200/_cat/indices?v&s=index" -O closedindices.bat
vim -s vimclose.vim closedindices.bat
closedindices.bat

For Linux

wget "http://10.89.78.67:9200/_cat/indices?v&s=index" -O closedindices.sh
vim -s vimclose.vim closedindices.sh
bash closedindices.sh

Failed to validate file names - Azure Storage File upload

When I am trying to upload a file in Azure Storage, I got an error, "Failed to validate file names". The file name does not have any special characters or unicode characters.

The actual reason is, my IP is not whitelisted for that storage account. When, I whitelisted my IP, I could upload the file without any issue.

Event AzCopy command also did not work when my IP was not whitelisted. At the same time, it did not show any reason on why the request failed.

Software Performance Improvement

Whenever anybody asks me to improve performance of any software, typically, I say I cannot do it.

But, what I do is,

Clean the dirty code.
Remove the redundant code.
Refactor at method level to make it more clean and simple.
Refactor at class level to make it more clean and simple.
Make the design more clean and simple.
Change the design to make it more simple.

If this improves the performance, then well and good. After all this, if the performance is still not improved to the expectation, then I would say, this is the limit. If you want better performance, get better hardware or get more servers.

For performance improvement, if anybody does any of the following, I try my best to stop them from doing it.

Any change to the code/design which reduces the simplicity of the code/design. This is one of the biggest grey areas and it is mainly trade-off. How much are we ready to sacrifice the simplicity/cleanliness of the code for how much performance. Typically, I am always on the extreme side of clean code/design.

To improve the performance of some functionality, which is there in the performance test suite, reducing the performance of other features which are not there in the test suite.

Not caring about the functionality which is having very bad performance, since it is not there in the performance suite, and caring about only the features that are there in the performance suite to improve the numbers.


If anybody gives me clean code with clean design, and asks me to improve the performance, then they would say that, I am the most incompetent developer that they have ever seen.

Time Complexity

If I am an organizer of Kaun Banega Crorepati, I will ask the following question as the last question.

What is the complexity of this? for(i=1; i<n; i++) print i;
A. Logarithmic
B. Linear
C. Polynomial
D. Exponential



If I am giving input, and you are calculating output, then

Linear - You spend roughly equal amount of time as me
Polynomial - You spend significantly more amount of time than me, but, you will do it.
Exponential - I spend couple of mins, and your whole life is not enough to calculate the output.

My input is 100000000000000000000000000000000000000.

Taking Screenshots From Secure PDF/Adobe Digital E-Books

If you want to take the screenshots of Secure PDF or E-Books that can be opened only in Adobe Digital Editions, you can use the following software. It is very simple to use.

Open your PDF/E-Book and change the size in such a way that, on click of Page Down, it goes to next page.
Run the software
Select the Program (Adobe Reader/Adobe Digital Edition)
Click on Select Area and select the page area which you want to capture.
Select the folder and file name prefix.
Click Take Screenshot button. It takes the screen shot of the selected area and goes to next page.

One disadvantage with this is, if there are 100 pages in the PDF, you need to click Take Screenshot 100 times.

Executable - http://www.tnsatish.com/softwares/TakeScreenshots.exe

Source Code - http://www.tnsatish.com/softwares/TakeScreenshots.zip

Time Complexity - Linear, Polynomial and Exponential

Whenever I discuss about the complexity of programs, most of the times, I drain all my energy in few minutes (because of talking in very loud voice). The following example is very famous.

Is it linear, polynomial or exponential?

calculate(int c, int n)
{
   BigInteger x = 1;
   for(int i=0; i<n; i++) {
       x = x*c;
   }
   return x;
}

If I am giving input, and you are calculating output, then

Linear - You spend roughly equal amount of time as me
Polynomial - You spend significantly more amount of time than me, but, you will do it.
Exponential - I spend couple of mins, and your whole life is not enough to calculate the output.

If anyone says the above program is linear or polynomial, I am ready to spend 10 seconds of my time in giving the input. Are you ready to calculate the output?

What is Test Driven Development?

The developer develops the bare minimum functionality for the entire task without any unit testing and delivers it to the Test team.

The Test team tests the functionality and files bugs saying either the functionality is missing or not working for mainline scenarios.

For functionality missing bugs, the developer implements just the mainline scenario, and for the main line scenario not working bugs, he makes it work just for the mainline scenario and delivers the code.

The test team tests the functionality again, and files bugs saying, either the mainline scenarios are not working for the previously missing functionality or other scenarios are not working.

Developer fixes those again, and delivers it.

Then the test team finds few more scenarios and few corner cases and files bugs on those.

This cycle is repeated, till the code satisfies all the requirements.

This is called Test Driven Development.

The advantage in this method is, the developer does not need to give accurate estimates to the manager, in terms of no.of tasks or duration of each task. We can finish the project, whenever the manager wants. [i.e., we say that, we are done with the tasks on the deadline given by the manager.]

The only disadvantage is, the testers will lose interest in software and/or their life.

[If you really don't know what is test driven development, and if you follow this, then the testers' life becomes miserable. In real test driven development, we have to replace test team with automated testcases.]


With sympathy on all the testers who are testing my code.

There is only one reality. That is Code. Everything Else is Illusion

There is only one reality. That is Code. Everything else (design and documentation etc.) is illusion.

Once I worked in a team, where the managers expected detailed design for everything. When, I say, we will have a single library for validation and data storage etc., they won't accept it. They expect the application to be modularized, and have many components. If somebody proposes multiple libraries, with each library doing one small work, they would be very happy about it.

I am very bad at presenting these things, and coming up with big architecture for very small problems. There was a person in my team, who is very good in these things. He came up with a design, in which, there are multiple libraries. One library for business logic and one for data storage etc.

There will be one business logic library, which does all the validations and processing of the data and the conversions from UI to DB and vice versa. The DB layer handles all the save and retrieval of data, and it is not burdened with any processing of the data or business logic. The entire business logic would be in the business logic library. In future, if the business logic grows, we can split into multiple libraries based on the functionality. Since, it is the first version, we can start with one library and see in the next release whether to split this library.

At the end, most of the methods in the business logic library are like this.

public BIObject GetBIObject(Parameters)
{
    return dbHandler.GetBIObject(Parameters)
}

public BIObject SaveBIObject(Parameters)
{
    return dbHandler.SaveBIObject(Parameters)
}

[For those who did not understand the above, the entire functionality (validation, storage and everything else) was developed in the data library. The business logic is just a wrapper with one line of code for each method.]

Everybody was happy by this method.

The managers were happy, because, the design is good, where there are multiple components, and it is modularized. The other developer was happy, because, the managers were impressed with his design. I was happy, because, the code was exactly the way, I wanted it to be. And, also, I was relieved from the burden of designing (which I never used anyway) and presenting it to the managers.


Related Links:
What does it take to convince a Manager?
Developing Libraries - Give Minimum Privileges

Pair Programming

In any big project, the people in that project would be having different skills at different levels. The skill set can be different in design, coding, performance optimization, or the knowledge of the existing functionality, code base, coming up with scenarios etc.

When the developers are working independently, if they don't have all the skills, then the feature may not be complete in all the areas. They may implement the functionality, but, it may not scale, or, they may come up with good design, but, misses many scenarios. In those cases, typically what happens is, two ore more developers would be working on the feature one after another. After the first developer implements with the skills that they have, second developer takes over and fixes the issues. If one developer implements the functionality without any performance, second developer may optimize that for performance. If one implements with good design and provides good framework, another one may add all the scenarios. In most of these cases, the second developer would be re-writing what the first one had written. After re-writing, the first one would review it, to make sure, they did not lose anything in re-writing.

If both the developers sit together and implement the functionality, then it would take less time than both implementing individually, one after another. When they do it together, they would be covering all aspects first time itself, and it would save significant time.

Pair programming does not work well, if one is having very good skills in all areas when compared to the other. In that case, it would become like, one dictating the code and the other typing, or one typing and other just watching. It works very well, when both have skills in different areas or almost equal skills in all areas.

I have seen re-writing of the code by different people many times in the name of review, bug fixes or improving performance. Many times, it could have been avoided, if they had thought about the solution together.

Eventhough, Pair Programming is a good practice in general, it is not followed by many. When two people work together, the immediate question that is asked by many people is, why cannot one person work on that? It is difficult to answer, because, it is very difficult to tell something like, "I don't know ABC's of the existing functionality, and the other developer does not have good design skills". It is almost impossible to tell this to the manager. If the developers are of opposite sex, then they may face different issues.

Pair programming may miserably fail, if the programmers do not have good relationship or their thinking is not in sync. Of course, if the programmers do not have good relationship, not only pair programming, but, even normal development also would be effected. The following is a snippet from an article in yourstory.

If you have two lead architects designing a system, you’re likely to end up with a two-module project. This phenomenon was first observed in the 1970s, when compiler design teams with 4 lead engineers ended up with 4-pass compilers. If your team had only 3 lead engineers, they were likely to design a 3-pass compiler. The design of the interfaces between the modules is also likely to be heavily influenced by the relationship between the lead programmers. If your system has 2 architects that are good friends, you’re likely to end up with a two well defined modules with clean interfaces and detailed contracts. If the two are not very good friends, they’re likely to come up with a design that relies too heavily on standard interfaces (like XML/WebServices) that may ultimately not be flexible enough for the system.

Questions/Suggestions for which I don't have Answer

• Are you still writing code? Are you not a Project Manager? Not even a Project Lead? Many of our friends are Project Managers now. I thought, you would be in a bigger position than us.
  
• Don't fight with your leads and managers. Do whatever they say. Once you become lead, then you can do whatever you want.
  
• Don't stay in one company for long. Change companies regularly. Otherwise, you will not grow.
  
• Why are you not trying for US?
  
• Why did you resign when you don't have job in hand? (Few people in my absence say that, I was fired.)
  
• Why don't you apply to big companies like Infosys, TCS and Wipro?
  
• Why did you resign, when you had to pay couple of tens of thousand rupees (for giving short notice)?
  
• Why did you change job, when you are not getting any hike?
  

I Don't Want to Work on Bug Fixes

Many people think that, fixing bugs is inferior work, and many people try to avoid it.

Many people forget that, the quality or greatness of the work is not in the work itself, but, how the people make out of it.

Recently, I got an interesting bug. The bug is very simple. It is a Null Pointer Exception. If that bug is given to any fresher, who has little programming knowledge, they can fix it in five minutes, and it will pass the testcase. But, I took lot more time.

When I had written the code, I assumed that, it won't return null, and coded accordingly. Now, when I got null, I checked why it is returning null. I found that, the corresponding Dictionary has not been populated. It was not populated, because, DB was not returning it. When I checked, what DB is returning, I found that, when the input has 600 ids, DB is returning only for 200 entities. The issue is, the stored procedure's input parameter can hold only 200 ids. Increasing the size of the input fixed this issue.

But, this is not the final solution. The null exception is coming even for basic scenarios. In basic scenarios, it should not be processing hundreds of entities. When I checked, why it is processing that many entities, I realized that, at one place, where it has to update it's dependents, it is updating all the entities in that component. When I am creating a new entity, it won't have any dependents. But, since, it is taking all the entities in the component, it is trying to update all the entities in the system.

My final fixes were, changing the code to update only it's dependents, and increasing the size in the stored procedure. I did NOT add null check in my code. If I get null again, it means, there is some big issue somewhere else.

If I had just added null check in my code, it is very difficult to find the issue, because, this entity would be updated properly. This failed in an asynchronous job. So, it is very rare that, as part of this testcase, tester tests other entities which are not involved with this entity. While executing the testcase for dependents, if the component does not have 200 entities, then that also would pass.

There is no work that can be done by the less qualified people. It is just that, we do it like the way less qualified people.

By the way, since, that Null Reference exception came in my code, I did more investigation. If it had been in other's code, even I would have finished it in five minutes. ;)

On a different note, my lead told me to put the effort spent for bug fixes in hours and not in minutes. I don't know how he felt, when I said, I am putting effort in hours only.

Design of Web Service Vs Performance of Client

Long back, I was working on a desktop application which handles lots of data. That application connects to a web service to get the data. Once the data is downloaded, the user works on the data offline, and then uploads the updated data.

One fine day, I got a mail from the web service team that, I was using one web service call which gets the data for all the entities in a given component. They were planning to remove that web service, and asked to me to use two other web service calls. First web service gets me the ids of all the entities in the given component, and the second web service gets me the data for a given entity. So, I need to call the first web service and gets the ids of all the entities, and for each id, I need to call the second web service to get the data. If there are 100 entities in the component, then there would be 101 web service calls. The reason that they gave for this change was, the web service was taking lot of time, and taking lock on SQL Server for a long time, which is blocking other transactions.

As usual, I did not agree with their proposal. My arguments were,

• There will be too many web service calls from the client to the server. Performance of the client would go down.
  
• The performance of the server also would go down. If they have single stored procedure/web service, the server would do much less work than 100 web service calls/stored procedures.
  
• If they are worried about SQL Server taking locks, they can have another wrapper web service which does all this work, and send it to the client, so that, client does not need to worry about the changes in the server.
  

Their arguments were,

• The stored procedure for the original web service is very expensive, and taking huge time to execute. Till then, it is holding the lock and other transactions are blocked till that time.
  
• Eventhough, with the proposed changes, the performance of this operation would go down, but, overall performance would increase, because, other transactions would not be blocked for a long time.
  
• All the web service calls map to a stored procedure in the DB. Creating another web service which does not have any corresponding stored procedure does not look good. This would be the only web service call that would be odd, and they do not want to pollute the web service.
  

Till, I was in that project, I have not done what they had asked. I don't know what happened, after I moved out of that project.

Who is wrong in this case?

[Of course, even if you say that, I am wrong, I may not accept. Because, I still feel that, Webservice should be driven by the requirements of the user rather than the DB design.]

Discrimination Based on Sex, Caste and Religion

Two people (one guy and one girl) contact me once in a while, for any help required in their application.

Whenever the girl asks for help, it is done in 5 mins to 5 hours. Only once, it went beyond that.
Whenever the guy asks for help, it is done in 5 days to 5 weeks. Only once, it is done in less than 5 days.

Nobody has criticized me directly on this. But, it is quite possible that, they think, I am discriminating based on sex. 

Whenever the girl sends me an email for any help, she puts the entire information that she has in the email. If the information is huge, she dumps everything in a document and attaches that document.

Whenever the guy sends me an email for help, he puts enough information required for any user of that application. He does not put any redundant information and does not attach any extra documents so easily.

If I have any follow up questions, both would respond within one working hour.

In general, if there is any attachment, generally, I won't open it immediately. I would open it later, whenever I am free. Even by this logic, I should be more responsive to the guy rather than the girl.

Unfortunately, people underestimate/ignore many many small things which can change everything. The small things matter a lot in the real world, and if everybody concentrates only big things, we can never develop.

I am very lazy and busy. I am neither a user nor a developer of that application and I don't know the technology that is used in that application. It is developed by a less qualified person, and I am just maintaining it. If I want to do even small change in that, I have to spend lot of time for that. In such a situation, whoever is asking for help, I would be happy to get as much data as possible. When the guy sends the information, he sends enough information for the user of that application. But, I am not a user. So, I need more details. I had to follow up with that guy multiple times.

How exactly one can go to the screen that he is referring? Once I reach that screen, which user should I chose to test? What all operations can I perform on that application so that, their data is not modified? These are all obvious for any user of that application, but not for me. So, I had to follow up for each of those things one by one, because, I discover the second one only after the first is clarified. I generally work on these things at night or weekend, and he won't be available immediately to answer at that time. So, I get the response next working day. Each follow up would increase the time significantly. The general tendency is, if I had to spend more time and follow up many times, then I would delay it more. For example, if I followed up for four times, for first follow up, I may reply in one day, whereas for the fourth follow up, I may reply only after a week.

Once that task is done, after few days, he asked me to do some other change at some other place. By then, he deleted the test user. So, I had to follow up again to create the test user. After doing that, then he said, the same thing has to be done at other places. If he gives the entire information with detailed steps on everything, and asks me to solve it, then, I can spend least amount of time and get it done.

First time, when the girl sent me a big attachment with so much information, I got bewildered. But, when I started doing that, I never needed anything extra, and in one single sitting, I finished everything that she asked. Even for small things like, application is down, she sends full error message that she gets, eventhough, I could find it myself. That would save me few seconds, and I can directly login/open appropriate application/server to fix the issue.

Whenever she sends any mail to me, she does everything she could do, and spends a lot of time in composing the mail with all the information. She communicates to me in such a way that, my time is very valuable, and she tries to use minimum time of mine. Whereas, it is the exact opposite with that guy.

I respond faster to those who give more value to me and my time. I also expect, if they want to solve their problems, they should have more interest in solving their problems than me. Otherwise, I won't respond that much faster.

In towns and villages, many times, people of different castes talk in different ways with others. Few caste people may talk to others always in singular, and few other caste people talk to others always in plural. If somebody asks me a favor without giving respect, I would show minimum interest in solving his/her problem. But, if they give minimum respect, then I may spend time to solve their problems. If only one particular caste people give me respect, then it would look like, I discriminate based on caste, and help only that particular caste people.

I discriminate based on how much the other person values me and my time. If only particular sex, caste and religion people values me and my time, then one can assume that, I discriminate based on sex, caste and religion.

Google - Best to Worst

At one point of time, Google was the best in everything, and there was no feature that could have been developed in a better way. I don't know, what happened in the recent time (don't know whether the changes in the top management effected this or not), Everything is becoming worse, and one had to continue with google, since there is no other competitor for most of the things.

Gmail - The new interface is very bad. I don't know who is the new program manager for Gmail. It seems, he/she gets one or two mails per day. That's why, he/she might have designed that way. The old program manager designed gmail for heavy load. They might have fired him/her to save the cost.

Google Reader - All comments of gmail are applicable to reader as well. They removed sharing feature to promote plus. So, those who read others' shares have to visit plus to read. Even then, it won't share the entire post. It would share only the first few lines, and the user has to visit the original site to read it.

Blogger - Initially, they made the interface worse for the advanced users. Now, they changed it to make it usable for advanced users also. Still, In the new interface, they have given more preference to icons rather than text. If I see New Post and the corresponding icon, I always would act faster for the text rather than the icon. [Don't know, whether some doctor would call this as a disease or not.]

Buzz - I don't know what made them to kill Buzz. But, it was the most simple interface, and only because of that, visitors to my blog has increased by many times.

I am still using the old interfaces for gmail and blogger. I could not find any way to use the old interface for google reader.

Security - Previously, Google has given the maximum preference to the security of the user's accounts. Now, it seems, they are least bothered about it.

Android OS - (Tested with Samsung Galaxy i5510). When you login to the gmail account from mobile, it stores the password. One cannot sign out, and removal of the password is also not possible. One has to do restore to factory settings to actually remove it. User can login to a dummy account, and from there, they can add another account, by which they can login to gmail and others. When they are done, they can remove that account. But, the problem is, till, you remove that account, it stores the password. If you lose the phone, when the account was added, they would get the access to the entire account. One has to change the password in that case. Or one has to enter the password, everytime, they want to access their account, and remove the account, once they are done. It is too tedious for anyone who worries about security.

There was one old gmail app from google, which works in Nokia, that does not store the password. It stores only the cookie. So, the user can access the gmail account at any time without entering the password. Since, it stores only cookie, we can remotely sign-off, in case we lose the phone. But, I cannot find that app now.

Another worst thing is, once you change the password, you cannot reuse the old password.

Google App Engine:

To upload any application in google app engine, you cannot use a browser. You have to upload it from command prompt/eclipse or other tool. When you use command prompt, while uploading, it asks for the username and password of the gmail account. Once you enter the username/password, anyone from that system/login can update the same application without entering the password again. It stores the cookie in the system and google never says where the cookie is stored. The cookie would expire after the time period that is set in the settings of the google app engine. The time period varies from 1 day to 2 weeks.

In windows, it stores the cookie and email id in Registry in HKEY_CURRENT_USER -> Software -> JavaSoft -> Prefs -> com -> google -> appengine -> tools -> admin

In Linux, it stores in ~/.java/.userPrefs/com/google/appengine/tools/admin/prefs.xml file.

If we delete those, then while updating the application, it asks for username and password again.

Credit Card details:

If you are having paid version of any service from gmail, then to renew it, you have to store the credit card, or should be ready to bring down your service for some time.

Any service provider would give option to renew their service before the due date. But, google does not give that option. You have to store your credit card details in the account, and on that specific date/time, it would charge your credit card. If your credit card does not work on that date for any reason, your service would be disrupted. If you do not want to store the credit card at all, then one day before, you have to cancel the service, and immediately take the service. But, during that process, the service may be disrupted.

Google says the data they store is secure. But, they made everything else as very insecure. For those who do not know how the google authentication/security works, if they login to google account from android mobile, or login to app engine or give their credit card details in a shared gmail account, then there is no security to their account/credit card.

Since Google is indispensable, and there is nobody in the competition, we have to live with these. There is no other option.

Thanks to Michael and Rajesh Tiruveedula

Different Roles in Software Development

In many product based companies, the developers at different levels starting from Trainee Software Engineer till Distinguished Engineer, are given the same kind of work. But, the way they do that work would be lot different.

If some work is given to developers of different levels, and if all of them do their best, then others may say the following in response to their work.

Distinguished Engineer - He has not delivered anything.

Principal Software Architect - He has not written even single line of code.

Technical Architect - Even a kid can write the code that he has written.

Sr.Software Engineer - He has developed a complex functionality single handedly.

Software Engineer - He has developed one of the most complex functionalities with very good design, which uses all the concepts of design patterns.

Trainee Software Engineer - He has developed the application by using all the latest technologies, it's latest features, and all the concepts of design patterns. To understand what he has written, one needs to learn all the latest concepts in the latest technologies.


(If the problem can be solved by changing the process or the way they use the system, then it won't require any new product to solve the problem. Even if it cannot be solved by changing the process, if it can be solved by using existing products in a different way, that would be better. That's how, Distinguished Engineers and Principal Software Architects work.)

By the way, My lead is taking lot of time to understand the code that I have written with the latest concepts. I also used the following comment, while sending for review.
// When I wrote this, only God and I understood what I was doing
// Now, God only knows

When do we get time to clean our code mess?

When do we get time to clean our code base?

This was the question asked by one of my colleagues to the Executive Vice President in the all hands meeting.

The Executive VP heard it like code mess (and another VP corrected it). He said something like, come up with a plan with the manager and the director, and he would consider that etc.

The following are my thoughts on the same.


Before asking the time for cleaning up the code base, I have few questions.

Did you write clean code yesterday?
Are you writing clean code today?
Will you be writing clean code tomorrow?

If the answer is No, then there is no point in asking extra time to clean up the code base. If we are not going to write clean code in our every day tasks, and asking separate time to clean the code base, there is absolutely no use, and it is just waste of the time, resources and money. First we should concentrate on writing clean code everyday, and then we should think of cleaning the existing code.

In any business, it is of no use to spend some extra time to clean some code base, unless we are 100% sure that, the code need to be changed soon with short dead lines, and also before that, we will not get any feature which requires re-writing of that. If the code is going to be re-written soon, then there is use of cleaning it now. We can directly re-write it later on. Whenever we get the change request in a component, at that time, we can try to fix the design/code of that component. Only, if we are sure that, the time that is given to those change requests is not enough with the existing design, we should fix the design now. Otherwise, there may be no use of cleaning it now.

If we are modifying a component, then the modified design should be in such a way that, all the parts that you are touching and it's integration with other parts would become very clean after that. If you are writing code, not only the exact parts that you are touching, but, all surrounding parts of the code also should be cleaned.

If you are changing a method as part of a bug fix, then may be you should look at the entire method and it's usage, and see whether you can rewrite the method in a very efficient way. If you are adding a method in a class, may be, you should look at redesigning of few methods, so that, the class would become clean.

Whenever any developer changes any code, if they look at the high level picture, and modify the code/design accordingly rather than just implementing that feature alone without bothering anything else, then within couple of months/years (depending on the size of the project), the entire code base becomes very clean.

Generally, no business person would give dedicated time to clean the code, unless, they are 100% sure that, the code would be maintained a lot and will not be re-written. They may also give time, if they have deep pockets. If they have deep pockets, probably, you can ask for salary raise first.

Courtesy: Shawn Smith (Who has shown me the difference between fixing a bug by a developer and a Distinguished Engineer. If he fixes a bug, then only another distinguished engineer can fix in such a clean way).

How Different Companies Respond to New Requirements

Apple

You don't need that functionality. We know what you need more than you. Don't argue with us. You don't need that functionality.

Open Source

That functionality is already implemented. You cannot come up with any requirement, which we have not yet implemented. May be they are just command line tools and may not have UI. Most of the companies just develop the UI on top of our libraries and sell as if they developed it everything themselves.

Microsoft

Nobody is asking us the requirements. Everybody is just hacking our software, and developing whatever they want. When we release newer version, and if those hacks do not work, then they just blame on us saying that, we don't maintain the backward-compatibility. The open source enthusiasts criticize us that, Linux is more windows friendly than Windows Vista, eventhough the fault is with those who hacked our softwares and developed applications with unsupported, undocumented, undisclosed internal APIs.

Google

We give 99% of the functionality required by 99.99% of the people. If you are in the minority, pray God.