You are not the highest bidder - but you were offered the auction item

You participated in an auction. You did not win the auction. After few days, the auctioneer called you and offered the auction item for your highest bid. Would you buy that item for your highest bid?

That might be a potential scam. 

You are interested in buying an item for Rs.20 Lakh. The next actual buyer is ready to buy for Rs.10 Lakh. Let's say, in the auction, the minimum increment is Rs.1000. In a fair auction, you can get the item for Rs.10,01,000. 

Let's say, the auctioneer put someone from his team to win the auction. Then, that person wins the auction for Rs.20,01,000. Since, the winner is from the team of the auctioneer, they won't actual take the item. After few days, you would get call from the auctioneer that, they are ready to offer you for the amount you quoted. If you accept that, you are going to lose big time. 

Even if you ask for another auction to be performed, it is not going to help that much. Because, the auctioneer knows that, you were ready to buy for Rs.20 Lakh, they can manipulate you to buy for atleast Rs.18 Lakh or Rs.19 Lakh by having their own member in the next auction as well. 

When someone has done a scam like above, and if they are doing re-auction, it is not going to be fair by any means. Because, the auctioneer has the complete data of everyone's highest bid. They can easily manipulate it, so that, they can get a good deal. 

NPCI asking Google Pay/Phone Pay to reduce their transactions

NPCI has asked Google Pay and Phone Pay to reduce their number of transactions, since they are becoming monopoly in the UPI transactions. 

Google Pay and Phone Pay have share of almost 40% each, and NPCI asked them to reduce their share to 30% (Of course, they are given some time to reduce their share). 

It is one of the most stupid rulings by NPCI in the name of Monopoly. It is like going back to Pre-1991 days, where Nehru-Gandhi family dictated each organization how much they could produce. After producing their licensed quota, they need to stop the production, even if they are capable of producing more and have enough money and resources. 

In general, to break the monopoly, the concerned authority should not do anything except giving license to everyone liberally and making sure the organizations are following the rules set by them. 

In this case, NPCI has already given licenses to many organizations. So, there is nothing that it should do. How much business would the organization get depends on their quality of service and the customer satisfaction. NPCI should not put its finger in that. 

If NPCI wants to break the monopoly, it should go and ask more organizations to provide UPI services and give even more licenses liberally. However, it should never discourage anyone (like it is doing in Google Pay or Phone Pay case). 

The only other thing that NPCI should worry is, breaking the rules set by NPCI. For example, NPCI may say, the application should transfer to any UPI. If Google Pay or Phone Pay are intentionally failing transactions to UPIs of other providers, then that can be taken as non-compliance and can impose penalty for that. The penalty can be huge so that they follow the rules properly. But, they should not ask them to reduce their share in the market. 

Another mistake that few authorities are doing is, imposing monopoly restrictions because of the offers that big players are giving. 

The big organization (which is having a lot of money) gives very good offers to the public [The service is sold at less than their cost]. Everyone starts using the services of big organizations. The small organizations cannot sustain and in due process, those would be closed. Once most of the small organizations are closed, the prices are increased significantly and the big organization gets enormous profits. 

With this logic, authorities try to limit the offers given by the big organizations. But, authorities should never try to do that. They only need to make sure that a new organization can be started for providing that service without much hassle, and people have the option to switch the provider. 

Many years back, Foodpanda used to give a lot of offers and people used to buy food from that. Once they stopped giving offers, people stopped buying from that, and people simply switched to other providers. If any authority had asked them to reduce the offers, the public would have lost a lot.

When Paytm started, it gave a lot of offers. After demonetization, it reduced its offers and at the same time, Google Pay started with a lot of offers and people shifted to Google Pay. If a new provider starts giving offers, then people won't take any time to move to that new provider. 

The goal of the Union Government is to increase digital transactions. When Google Pay and Phone Pay are penetrating a lot to reach more people, the Government should not put any barriers on that. 

NPCI is not run by the government. But, why is NPCI given such power to reduce the business of the organizations?

Elasticsearch - Backup and Restore using Snapshots

The data in Elasticsearch can be backed up using snapshots and it can be restored to a different Elasticsearch cluster.

To take backup of the Elasticsearch on the source cluster, first we need to create a repository (in Elasticsearch) with the target location and type.

The type of the repository is, where do you want to store the backup files. The type can be File Share, Microsoft Azure, Amazon S3, Google Cloud or Hadoop HDFS.

Complete details on Creating a Snapshot is found at https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html

Creating a File Share Repository:

put http://10.21.32.43:9200/_snapshot/mybackup

{
  "type": "fs",
  "settings": {
    "location": "/data/elasticsearch/backup"
  }
}

The above request would create a repository of type File Share and the backup files would be copied to /data/elasticsearch/backup

Creating a Repository in Azure:

Complete details of Azure Repository can be found at https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-azure.html

Before you create a repository in Azure (or even for other providers), please check your access to Azure Repository from your local and from all the Elastic Search Servers. Admin can configure access to cloud storage based on IPs. When there is an IP restriction in Azure, there is no way to find out that from the error messages.

We have to install a plugin for Azure for backup/restore. Command for installing azure plugin is

bin/elasticsearch-plugin install repository-azure

In linux, you may have to use sudo (If not running from root account). elasticsearch may be found at /usr/share/elasticsearch/bin

After installing Azure plugin, we need to add the storage account name and the secret key in Elasticsearch to connect to Azure.

echo This/is/a/key/for/Azure/Storage== | elasticsearch-keystore add -stdin azure.client.default.key

echo AzureStorageAccountName | elasticsearch-keystore add -stdin azure.client.default.account

You may have to use sudo in linux.

Once Elasticsearch is configured to use Azure, we can create a repository.

put http://10.21.32.43:9200/_snapshot/mybackup

{
  "type": "azure",
  "settings": {
    "client": "default",
    "container": "test",
    "base_path": "testpath"
  }
}

This would create a repository, and the snapshot files would be created in testpath in the test container of the storage account.

Taking Snapshot:

The below request starts the process of creating a snapshot

put http://10.21.32.43:9200/_snapshot/mybackup/snapshot1?wait_for_completion=false

With the below request, you can see the status of the snapshot. For any snapshots in progress, it would show the details of the snapshot like how many indices are completed, in progress, not yet started etc. If there are no pending snapshots, it would return an empty array.

put http://10.21.32.43:9200/_snapshot/_status

Incremental Snapshots:

When you take multiple snapshots in a repository, each snapshot is complete for all practical purposes. Any snapshot can be restored or deleted independent of any other snapshot.

But, internally, the snapshots are incremental. i.e., when you take another snapshot in the same repository, it would create the files for the changes that happened since the last snapshot. The metadata of the snapshot would be created appropriately to get the data required from that snapshot.

Creating Read Only Repository (On Destination):

On the destination Elasticsearch cluster, we need to create a read only repository pointing to the same location (either file share or Azure or any other cloud provider). Making the repository as read only is the only difference with respect to the creation of the repository on the source cluster.

put http://10.89.78.67:9200/_snapshot/mybackup

{
  "type": "azure",
  "settings": {
    "client": "default",
    "container": "test",
    "base_path": "testpath",
    "readonly": true
  }
}

Restoring the Snapshots:

The below request would restore the snapshot on the destination cluster.

post http://10.89.78.67:9200/_snapshot/mybackup/snapshot1/_restore

Restoring Status:

There is no specific request to check the status of the restore. However, when the restore is started, the cluster would go to yellow state. Once it goes to Green state, we can understand that the restore is completed.

The below request gives the status of the cluster.

get http://10.89.78.67:9200/_cluster/_health

Restoring Incremental Snapshots:

When you restore incremental snapshots, you would get an error since there is already an index in the destination cluster with the same name as the index that the restore is trying to create.

To restore the snapshot from second time onwards, you need to close the existing indices, so that the cluster can update them.

Request to close all the indices

post http://10.89.78.67:9200/_all/_close

Request to close one index

post http://10.89.78.67:9200/indexName/_close

Restore after Closing the Indices:

During the restore, Elasticsearch would open the indices that needed to be updated. If any index is not there in the snapshot, then that index would not be opened and it would continue to be in closed state.

If you are using the destination cluster only as a backup of another cluster, if you see any index in closed state after a restore, it means that index was deleted from the source cluster.

Handling Aliases:

If you are having aliases in the source cluster, and if the underlying index is getting changed and the old index is getting deleted, then you need to do special handling on the destination cluster.

For example, in the source cluster, you create one index everyday. The index name may be something like IN20201105, IN20201106 etc. You have an alias named INDate, which is pointing to that day's index. You keep only the active index and remove all the old indices.

In this case, when you take snapshots everyday, the indices are going to be different each day. The index that was present yesterday won't be present today.

When you restore the snapshots everyday on the destination cluster, the deleted indices would be in closed state.

When the indices are restored, the aliases would be restored with the relationship with the new index, but it won't delete the relationship with the old index. The alias would be pointing to both old and new indices. When you query the alias, it would have both the old index (that was deleted in the source cluster and closed in the destination cluster) as well as the new index. Since, one index in the alias is in closed state, you would get an error when you are trying to query it.

The simple option is deleting all the indices that are in closed state after a restore.

Request for deleting an index

delete http://10.89.78.67:9200/indexName/

Automated Snapshot and Restore everyday:

The snapshot can be taken in the source cluster without having any impact on the source cluster. However when you restore, the cluster would be down for some time.

You can use any scripting language that you are comfortable for the automation. I used WGET and VIM for automatic snapshot and restore.

There are 4 steps in the entire process of snapshot and restore. First step is on the source cluster and the next 3 steps are on the destination cluster. You have to schedule in such a way that, before the second step is started, the first step is completed.

1. Take Backup in the source cluster

Request in Linux

wget -d --method=put "http://10.21.32.43:9200/_snapshot/prodbackup/s`date +\%Y\%m\%d`?wait_for_completion=false" -O CreateSnapshotES.txt -o CreateSnapshotNetwork.txt

Request in Windows

In different versions of Windows (and/or localization), the date command is displayed with different formats. You need to check the date command's format.

wget -d --method=put "http://10.21.32.43:9200/_snapshot/prodbackup/s%DATE:~-4%%date:~4,2%%date:~7,2%?wait_for_completion=false" -O CreateSnapshotES.txt -o CreateSnapshotNetwork.txt

2. Close the indices on the target cluster

Request to close all the indices

wget -d --method=post "http://10.21.32.43:9200/_all/_close" -O CloseDRIndicesES.txt -o CloseDRIndicesNetwork.txt

3. Restore the snapshot on the target cluster

Request to restore the snapshot on the target cluster [Date part of the request should be same as in the request 1.]

wget -d --method=post "http://10.89.78.67:9200/_snapshot/prodbackup/s`date +\%Y\%m\%d`/_restore" -O DRRestoreES.txt -o DRRestoreNetwork.txt

4. Delete the closed indices

I used VIM and WGET for automating this.

Create a file vimclose.vim with the following content (Change the IP address in your script). Make sure that there is a new line at the end (after :wq).

:v/close/d
:%s/.*close *//
:%s/  *.*//
:%s/\(.*\)/wget -d --method=delete "http:\/\/10.89.78.67:9200\/\1" -O \1.json -o \1.txt/
:wq

Create a batch file or shell script with the following content.

For Windows.

wget "http://10.89.78.67:9200/_cat/indices?v&s=index" -O closedindices.bat
vim -s vimclose.vim closedindices.bat
closedindices.bat

For Linux

wget "http://10.89.78.67:9200/_cat/indices?v&s=index" -O closedindices.sh
vim -s vimclose.vim closedindices.sh
bash closedindices.sh

IRCTC - Autocratic, Unfair, Unethical and Monopolistic organization

IRCTC is one of the most Autocratic, Unfair, Unethical, Technically dumb and Monopolistic organizations in India. 

Since IRCTC is no longer a wholly owned government organization, Union Government should revoke the special privileges that it is getting. Till now, IRCTC is the sole organization which is selling tickets online. The Union Government should come up with a fair process and select multiple organizations to sell the tickets online. If the Union Government selects only one organization, it may do unethical, unfair activities and may trouble the people (what IRCTC is doing now). If the license is given to multiple organizations, it would encourage the competition and people would get better services. 

Also, IRCTC should not be having any special legal provisions which are not available to any other public limited company. IRCTC is using the RPF Acts to arrest others, when the mistake is with IRCTC itself. If IRCTC cannot secure their site, then IRCTC should not be allowed to sell the tickets online, and the license should be given to other organizations by following a fair process. 

Unfairly giving exclusive power to IRCTC (Selling tickets online)

IRCTC is a public limited company, and not wholly owned government company. Indian Railways is giving full power of selling tickets online to one single company, IRCTC. When it was wholly owned by the government, it was fine. But, when it is a public company, it is completely unfair. 

If Indian Railways want to give license to some organization for selling tickets online, it should come up with a proper process and invite any organization to bid for that, and the license should be given to multiple organizations that satisfy the criteria. [If License is given to one organization, that organization may become an autocrat.]

Indian Railways did not follow any of that process and gave exclusive license to IRCTC alone. This is unfair by any standard. This is worse than 2G or Coal scams. Media and Opposition parties are concentrating on the scams where black money exchanged hands. But, if we see the potential loss of money or unfair process on very big transactions, this is much more unfair than 2G or Coal scams.

Technically Dumb Organization

IRCTC is the most technically dumb organization. It is surviving solely because of the exclusive power given by the Government. If the Government stops giving exclusive power, almost nobody would buy tickets from IRCTC. 

IRCTC is blocking the users for using the autocomplete feature in their browsers. Autocomplete feature is there in the browsers for decades. Anybody who knows computers a little bit would use autocomplete to fill the forms. If IRCTC wants to block auto complete, it needs to come up with its own technology which makes auto complete not possible for the users. But, IRCTC did not come up with anything like that, but blocked users who used the browser's autocomplete feature. This is the dumbest thing by any standard, and unfortunately IRCTC does not understand that it is the dumbest thing.

Special Legal Powers for IRCTC

Since, IRCTC is a public limited company, the legal powers of IRCTC should be like any other public limited company, and it should not have any special powers. Right now, it is misusing the RPF Act and other provisions that are exclusive for the Government organizations. IRCTC is incapable of handling even basic (Software) security, but it is passing the blame to others and arresting them.

The following are the blatant misuses by IRCTC.

Auto Fill Software

Technically dumb IRCTC arrested one person in Vellore, Tamilnadu for using "Auto Fill Software". Anybody who understand software technology can clearly say, how dumb it is. Autocomplete feature has been there since the beginning of the browsers and many people use it. If IRCTC does not want people to use Auto Complete, it has to bring its own technology which prevents Auto Complete. Instead of doing that, it is arresting people or blocking users for using Auto Complete. 

It is like, IRCTC was thinking that it was in the 1700s (when there were no cars), and arresting people for using cars, bikes and other vehicles. 

Insufficient Security of IRCTC - Who needs to be blamed for theft?

If a person thieves from a house every day from the same house, who needs to be blamed? Is it the house resident or the thief?  A thief needs to be blamed. But, more than that, the resident of the house needs to be blamed. If a thief steals for one or two days from a house, we can say that the thief is wrong. But, if a thief is stealing from the same house everyday, it is the mistake of the resident of the house for not taking enough precautions even after facing theft multiple times. IRCTC is doing exactly like a resident. If anyone hacks their system regularly, then they should block the access for that hacker. Instead of that, they are putting the blame on the hacker. 

If a person has booked tickets unethically for a couple of months, what IRCTC is doing without fixing that issue. [In this statement, the word unethical is used only by IRCTC, and no person who has some technical knowledge would use that word.]

Intentionally Reducing Security - An Illustration

I have taken a few valuable items from others and told them that I will secure it in my house. One person is repeatedly coming to my house and stealing a few things from my house. I lodged a police complaint against the unknown thief and still the thief is stealing from my house and I did not do anything to secure my house. When others ask me to return their valuables, I will tell them that the thief has stolen everything and I don't have anything. Can I just live as if nothing happened? 

Any person who knows logic and has some common sense would say that, I am responsible for everything, and for the things lost, I need to buy that with my own money and return to the actual owners. 

In the same way, IRCTC is responsible for keeping it secure. If it is one off case, we can understand that, there is some issue. If IRCTC claims, someone hacked their system and booked X no.of tickets over a few months, it is the issue with IRCTC and not the hacker. Responsible people of IRCTC should be punished for non taking enough security.

Public Access - Private Access

There are two types of access to any website. One is public access. Other one is private access. Private access refers to the restricted access like access to the servers, databases and others which are not meant for the public. Public access is referred to the access that everyone gets without any permission. Typically, it means, anything that one can browse through the site. 

If anyone gets private access to the resources of any website, the website owner has to investigate how others got private access and should take necessary steps.

If anyone is getting something by the public access, which is not intended by the website owner, then it is the mistake of only the website owner and not by anyone else. Only the website owners have 100% responsibility.

As far as I know, I never read the news about getting private access to IRCTC by anyone. Even if it is there, most probably, it is due to some employee of IRCTC leaking the credentials or access etc. IRCTC filed most of its cases, where someone did something with public access. 

Public Access - Handling by Other Organizations

If the website owner does not want something to happen in their site, but, if someone could do it through public access, then it is the sole mistake of the website owner. The big organizations know that, and they offer a very good amount to those who find those mistakes and report to them. 

Awards by Major Organizations for Finding Security Loopholes

Microsoft - Rs.1.8 Crore

Google - Rs.1 Crore

Facebook - Rs.15 Lakh

Amazon - Rs.11 Lakh

IRCTC - Arrest people

Our great IRCTC arrests people saying they hacked their site. [For the cases filed by IRCTC, no good technical person calls it hacking. It is only advanced usage. Since, IRCTC is dumb in technology, it calls the advanced users' usage as hacking.]

Bypassing the Railway System

For one arrest of an individual, the media report says that the individual bypassed the railway system to book the train ticket. I don't know whether it is mentioned like that in the case report, or the media used its own words. There is absolutely no meaning to bypass the railway system to book a ticket, because the tickets are in the railway system. The person would not even have bypassed IRCTC. The person would have used some software technique to book the tickets faster (which a non-software engineer might not know). 

As mentioned above, if IRCTC does not want to use the advanced features in the browsers, they should come up with new technology, and not arrest people who are using advanced features.

Making Money Illegally was Crime

There is a mention that the person was charging Rs.10/ticket and it is illegal to make money that way, and it was a crime.

For example, if I am not in a situation to book a ticket myself, and if I offer someone Rs.10 to book on behalf of me, are you going to arrest the person who booked a ticket for me? If so, you need to arrest the majority of Indians. 

Penalty for unauthorised carrying on of business of procuring and supplying of railway tickets

The above sentence is mentioned in one media report of one person's arrest. If this is correct, Railway police needs to arrest the head of IRCTC and other top people of IRCTC. The reason is, IRCTC is not fixing the issue, and allowing people to exploit. Even when someone is exploiting, IRCTC is arresting that person and blocking that person's app, and not fixing the issue of IRCTC. That's why, repeatedly different people are hacking. So, IRCTC top people must be arrested for allowing others to exploit the railway system.

[Hacking and exploiting words have to be taken from the definition of IRCTC. No good technical person would call that hacking or exploiting.]

Can non-tech people book a tatkal ticket easily?

Before IRCTC says useless things like, "deprived of a chance to buy tickets", we have to check whether a non-technical person can book a tatkal ticket? I am sure, if you take a good technical person and non-technical person (who has enough experience in buying things online) in booking a tatkal ticket, the technical person has a better chance of getting a confirmed ticket. IRCTC is already depriving non-technical people from buying the tickets. It is arresting people who are helping others with their expertise. It is blocking the users who are using their expertise for themselves. 

Why is someone paying a higher amount?

IRCTC and other government organizations should introspect why someone is paying a higher amount for the same thing that is offered by the government organization for a lesser price. If Indian Railways or IRCTC are offering a ticket for a lesser price, then why are people going to others and paying higher amounts for the same ticket. Few people who do not have enough knowledge would say that, those people are blocking the tickets and the common man does not have any other option other than going to them and pay a higher price. This can be true for the printed government bus tickets, but not true for Train tickets (or even flight tickets). Indian Railways is very secure and a non-employee cannot block the tickets and sell it to others. 

Whomever IRCTC is claiming as hackers, they are just helping others to book tatkal tickets, because non-technical people cannot book the tickets. Few of them might be using automated tools to help others. Since, IRCTC is not helping non-technical people to book Tatkal tickets, they are going to someone else and paying higher amounts. 

If the same situation happens to any private commercial organization, they would solve the problem of non-technical people by some other means, rather than filing cases against people who are helping. In fact, it is a very big business opportunity for the organization. 

Change of name/age/sex in the ticket and other internal security issues

A person who is not an employee of Indian Railways/IRCTC cannot change the name, age, sex of any ticket. Whether IRCTC agrees or not, Indian Railways is very secure (By the definition of any popular technical organization). It is not possible for a non-employee to block the ticket and sell it to others. If I book a ticket, then the ticket can be used only by someone of the same name/age/sex mentioned in the ticket. If someone has changed any of those, it is definitely with the co-operation of some employee of Indian Railways/IRCTC. The employees of Indian Railways/IRCTC should be punished more, because they have internal access to the system. 

Share Price of IRCTC

IRCTC is having a higher share price, only because of unfair power that it has received from Indian Government. The moment the Union Government follows a fair process for Train Tickets, the share price would hit rock bottom.